Troubleshooting Tip: How to troubleshoot explicit proxy on a FortiGate-7000 or 6000 series

[fricci_FTNT]

Description

This article describes how to collect WAD debug logs on a FortiGate-7000 (or 6000) series chassis.
In a FortiGate-7000 (or 6000) series unit, the FIM (or MBD) is the slot responsible for load-balancing the data plane traffic and forwarding it to a specific FPM (or FPC), which will process it.

A single connection from an end user can be sent to any FPM (or FPC), and it may become difficult to properly capture all the WAD debug output for a specific source IP.

Scope

FortiGate-7000 or 6000 series.

Solution

In this troubleshooting scenario, a client with ip 10.10.10.10 is using an explicit proxy configured with IP 172.16.16.16 on the FortiGate and listening on TCP port 8080. The client is trying to visit the address www.example.com, which resolves to IP 2.21.72.72.

Create a clone of the explicit proxy policy which is handling the specific source IP traffic and place it above the existing policy ID. As the source IP, select the test client IP and enable the new policy. Note its policy ID, which will be used in the WAD debug later.

To better troubleshoot and capture WAD debug output for all connections from a specific source IP, a flow rule can be created so all TCP traffic from a specific user can be sent to a specific FPC. A flow-rule for each traffic direction must be created and enabled.

The procedure is as follows:

1. Create two flow rules:
   
   

   config load-balance flow-rule
     edit 0
       set status enable
       set ether-type ipv4
       set src-addr-ipv4 10.10.10.10/32
       set protocol tcp
       set action forward
       set forward-slot FPM4
       set comment "Test: proxy traffic from client to server"
     next

     edit 0
       set status enable
       set ether-type ipv4
       set dst-addr-ipv4 10.10.10.10/32
       set protocol tcp
       set action forward
       set forward-slot FPM4
       set comment "Test: proxy traffic response from server to client"
     next
   end

2. Two flow rules with a new ID will be created and appended to the list of existing flow-rules. To show all the flow rules on the screen run:

show load-balance flow-rule

3. Connect to the specific FPM (or FPC) using one of the two methods below:

• Recommended method: Open a SSH shell using Putty (or similar software), set the SSH port to 22XY (2204 for FPM4 on chassis-id 1) and log the output to a file (Technical Tip: How to create a log file of a session using PuTTY ).
• Alternatively, if it is not possible to use special management ports due to policy restrictions, connect to a specific slot using the following command from FIM (or MBD):

execute load-balance slot manage <slot-number>

3. Test if the packets are going through the slot configured in the flow rule (FPC06 in this scenario):

conf vdom
    edit <VDOM-name>
        diagnose sniffer packet any "host 10.10.10.10 and host 2.21.72.72 and proto 6" 4 30 l   <----- This will capture 30 TCP packets between client 10.10.10.10 and server 2.21.72.72, they should go through slot 4.

4. If the TCP packets are going through the expected slot, run the WAD debug using filters from the VDOM context:
   
   conf vdom
     edit <VDOM-NAME>
       diagnose wad debug enable level verbose
       diagnose wad debug enable category webcache
       diagnose wad debug enable category http
       diagnose wad debug enable category http2
       diagnose wad debug enable category policy
       diagnose wad debug enable category auth
       diagnose wad debug enable category session
       diagnose wad debug show  <-------this is to double check the filter settings
       diagnose debug console timestamp enable
       diagnose wad debug display pid enable
       diagnose wad filter vd <VDOM-NAME>
       diagnose wad filter dst 2.21.72.72
       diagnose wad filter explicit-policy <policy-ID>   <----- This is the policy ID specific for the test client IP
   
   
5. Once ready for the test, enable the previous debug output:

diagnose debug enable

6. Run the test and load the webpage. Curl can be used instead of a web browser: date; time curl -vvv -I -x http://172.16.16.16:8080 -L "https://www.example.com" ; date
   
   
7. Once the test traffic output is displayed, close the web browser window or terminate Curl.
8. Stop the wad debug:

diagnose debug disable
diagnose wad debug clear

In addition, it is useful to run a debug flo,w especially in case there is suspicion of packets being dropped:

conf vdom
  edit <VDOM-NAME>
    diagnose debug flow filter addr y.y.y.y   <---- -IP y.y.y.y is the IP address if the destination server.
    diagnose debug flow show function-name enable
    diagnose debug flow show iprope enable
    diagnose debug console timestamp enable
    diagnose debug enable
    diagnose debug flow trace start 99999   <----- It captures the first 99999 packets, then it stops.

Related documents:

7000E Special management port numbers 

7000E HA mode special management port numbers 

6000F Special management port numbers 

6000F HA mode special management port numbers 

Troubleshooting Tip: WAD troubleshooting commands 

Technical Tip: Gathering WAD debugs using the 'diagnose test application' debug command 

Troubleshooting Tip: Troubleshoot the explicit proxy in FortiGate 

Troubleshooting Tips: How to identify the policy ID number when using Explicit Proxy Policy 
Troubleshooting Tip: Example of wad debugging for Explicit Proxy