Technical Tip: HA Remote IP Monitoring

Anthony_E · ‎03-27-2008

Article

Description

HA remote IP monitoring (also called HA ping server) is similar to HA port monitoring. Port monitoring causes a cluster to failover if a monitored primary unit interface fails or is disconnected. Remote IP monitoring uses ping servers configured on FortiGate interfaces on the primary unit to test connectivity with one or more IP addresses of network devices. Usually these would be IP addresses of network devices not directly connected to the cluster. Remote IP monitoring can cause a failover if one or more of these remote IP addresses does not respond to a ping server.

Components FortiOS v3.0 MR6

Steps or Commands

By being able to detect failures in network equipment not connected directly to the cluster, remote IP monitoring can be useful in a number of ways depending on the network configuration. In the simplified example topology shown in Figure 1, the switch connected directly to the primary unit is operating normally but the link on the other side of the switch has failed. As a result of the failure, traffic can no longer connect between the primary unit and the Internet.

To detect this failure it is possible to create a remote IP monitoring configuration consisting of a ping server on port2 of the cluster. The primary unit tests connectivity to 192.168.20.20. If the ping server cannot connect to 192.268.20.20 the cluster to fails over and the subordinate unit becomes the new primary unit. The remote HA monitoring ping server on the new primary unit can connect to 192.168.20.20. As well, the new primary unit can connect to the Internet; so the failover maintains connectivity between the internal network and the Internet through the cluster.

Figure 1: Example HA remote IP monitoring topology

To configure remote IP monitoring

Enter the following commands to configure HA remote IP monitoring for the example topology.
config system ha
  set pingserver-monitor-interface port2
  set pingserver-failover-threshold 10
  set pingserver-flip-timeout 120
end
- Enter the pingserver-monitor-interface keyword to enable HA remote IP monitoring on port2.
- Enter the pingserver-failover-threshold keyword to set the HA remote IP monitoring failover threshold to 10. If one or more ping servers fails, cluster failover occurs when the priority of all failed ping servers reaches or exceeds this threshold. Set the priority for each ping server using the ha-priority keyword as described in step 2 below.
- Enter the pingserver-flip-timeout keyword to set the flip timeout to 120 minutes. After a failover, if HA remote IP monitoring on the new primary unit also causes a failover, the flip timeout prevents the failover from occurring until the timer runs out. Setting the pingserver-flip-timeout to 120 means that remote IP monitoring can only cause a failover after 120 minutes after previous HA failover. This flip timeout is required to prevent repeating failovers if remote IP monitoring causes a failover from all cluster units because none of the cluster units can connect to the monitored IP addresses. The "pingserver-flip-timeout" can be set in the range of 6-2147483647.
Enter the following commands to add the ping server to the port2 interface and to set the HA remote IP monitoring priority for this ping server.
config system interface
  edit port2
    set detectserver 192.168.20.20
    set ha-priority 10
end
- Enter the detectserver keyword to add the ping server and set the ping server IP address to 192.168.20.20.
- Enter the ha-priority keyword to set the HA remote IP monitoring priority of the ping server to 10 so that if this ping server does not connect to 192.168.20.20 the HA remote IP monitoring priority will be high enough to reach the failover threshold and cause a failover.
It is also possible to use the config global command to change the time interval between ping server pings using the interval keyword and to change the number of times that the ping fails before a failure is detected using the failtime keyword.
It is also possible to do the following to configure HA remote IP monitoring to test more IP addresses:

Enable HA remote IP monitoring on more interfaces by adding more interface names to the pingserver-monitor-interface keyword.
If the FortiGate configuration includes VLAN interfaces, aggregate interfaces and other interface types, it is possible to add the names of these interfaces to the pingserver-monitor-interface keyword to configure HA remote IP monitoring for these interfaces.
Add HA ping servers to other interfaces using the detectserver keyword to add the ping server and the ha-priority keyword to make the ping server an HA ping server and configure the priority of the ping server.
Add a second IP address to the detectserver keyword to monitor two IP addresses on each interface.

Note:

If adding two IP addresses to the detectserver keyword the ping will be sent to both at the same time, and only when neither server responds will the ping server fail.

Add secondary IPs to any interface and enter detectserver and ha-priority for each of the secondary IPs. It is possible to do this to monitor multiple IP addresses on any interface and set a different HA priority for each one.

Ping server priority and the failover threshold:

When one HA ping servers fails, its priority is compared with the failover threshold. If the priority is greater than or equal to the failover threshold, HA remote IP monitoring triggers an HA failover. If the priority is less than the failover threshold, a failover does not occur. If an HA remote IP monitoring configuration includes only one HA ping server, its priority should be the same as or higher than the failover threshold.

When more than one ping server fails, the total of the priorities of the failed ping servers is compared with the failover threshold. An HA failover is triggered only if the total of the priorities is greater than or equal to the failover threshold. If having configured two HA ping servers both with priorities of 10 and if the failover threshold is 20, an HA failover occurs only when both ping servers fail. If having configured three ping servers all with priorities of 10 and if the failover threshold is 20, a failover occurs if any two ping servers fail. And so on.

By adding multiple ping servers to the remote HA monitoring configuration and setting the HA priorities for each, it is possible to fine tune remote IP monitoring. For example, if it is more important to maintain connections to some remote IP addresses it is possible to set the HA priorities higher for these important IP addresses. And if it is less important to maintain connections to other remote IP addresses it is possible to set the HA priorities lower for these. It is also possible to adjust the failover threshold so that if the cluster cannot connect to one or two high priority IP addresses a failover occurs. But a failover will not occur if the cluster cannot connect to one or two low priority IP addresses.

Flip timeout:

The HA remote IP monitoring configuration also involves setting a flip timeout. The flip timeout is required to reduce the frequency of failovers if, after a failover, HA remote IP monitoring on the new primary unit also causes a failover. This can happen if the new primary unit cannot connect to one or more of the monitored remote IP addresses. The result could be that until fixing the network problem that blocks connections to the remote IP addresses, the cluster will experience repeated failovers. It is possible to control how often the failovers occur by setting the flip timeout. The flip timeout stops HA remote IP monitoring from causing a failover until the primary unit has been operating for the duration of the flip timeout.

If setting the flip timeout to a relatively high number of minutes , it is possible to find and repair the network problem that prevented the cluster from connecting to the remote IP address without the cluster experiencing very many failovers. Even if it takes a while to detect the problem, repeated failovers at relatively long time intervals do not usually disrupt network traffic.

Detecting HA remote IP monitoring failover:

Just as with any HA failover, it is possible to detect HA remote IP monitoring failovers by using SNMP to monitor for HA traps. It is also possible to use alert email to receive notifications of HA status changes and monitor log messages for HA failover log messages. In addition, FortiGate units send the critical log message Ping Server is down when a ping server fails. The log message includes the name of the interface that the ping server has been added to.