Technical Tip: Using the 'diagnose wad debug' command to troubleshoot Explicit Web Proxy related issues

Description

This article describes the 'diagnose wad debug' command and provides usage examples.

Scope

FortiGate.

Solution

To debug traffic proxied through the FortiGate, a WAD-related diagnose command has been added to FortiOS v5.6. The main purpose of this command is to get detailed info on client/server traffic that is controlled by the WAD processes.

The 'diagnose wad debug' command has the following main options:

diagnose wad debug ? 
enable                 <- Enable debug setting. 
disable                <- Disable debug setting. 
show                   <- Show debug setting. 
clear                  <- Clear debug setting. 
display                <- Display setting. 
save-http-req-crash    <- Save HTTP request when WAD worker crashes.

By default, the 'diagnose wad debug' command troubleshooting options are set to the following values:

diagnose wad debug show
Category: not set          <- There is no category set.
Level: info                <- The debugging level is set to informational.
Display: pid disabled      <- The PID display option is disabled.

Note: In order to start capturing WAD-related data, the category option must be set to something other than 'not set'.

To change the category, use 'diagnose wad debug enable category ?'.

The following options are available.

diagnose wad debug enable category
session       <- session.
packet        <- packet.
dispatcher    <- dispatcher.
http          <- http.
http2         <- http2.
cifs          <- cifs.
mapi          <- mapi.
socks         <- socks.
ftp           <- ftp.
icap          <- icap.
ssl           <- ssl.
webcache      <- webcache.
bytecache     <- byte cache
policy        <- policy matching.
auth          <- authentication.
scan          <- UTM scan.
tunnel        <- wanopt tunnel.
sys           <- sys.
video         <- cache video.
waf           <- waf.
memblk              <- memory block.
all           <- all category.

To change the debug information level, use 'diagnose wad debug enable level ?'. The verbosity of the debug output is increasing from 'error' (least detailed) to 'verbose' (full output).

diagnose wad debug enable level ? 
error      <- error.
warn       <- warning.
info       <- information.
verbose    <- verbose.

To display WAD Process ID information, use the 'diagnose wad debug display pid ?' command.

diagnose wad debug display pid ? 
enable/disable    <- Enable/disable PID display.

To start capturing data, select a category (e.g., http), then enable debugging using 'diagnose debug enable'.

diagnose wad debug enable category http
diagnose debug enable

The console output will look like the following.

wad_http_session_make(30455): make ok session=0x2a9a19d848 server=(nil) detect 10
wad_http_stream_get_line(976): http stream no line br_len = 225 i = 38 state 7
wad_http_request_reader_run(100): http reader 0x7fbffffab0 begin state=2
wad_http_request_reader_run(305): HTTP request method=4/7 version=3/8/0 uri=19/0 
bypass req(0x2a9a1ab508) caller(wad_http_init_req_status)@7908
wad_http_stream_get_line(976): http stream no line br_len = 187 i = 27 state 7
wad_http_stream_get_line(976): http stream no line br_len = 160 i = 30 state 7
wad_http_stream_get_line(976): http stream no line br_len = 130 i = 130 state 9
 [0x2a9a1ab508] Received request from client: 10.218.5.195:49582etc...

To stop capturing data, use 'diagnose debug disable'.

To display the WAD Process ID that is processing the data, enable the process ID display option.

diagnose wad debug display pid enable

To start capturing data, enable debugging using 'diagnose debug enable'.

The console output looks like the following.

913-wad_http_session_make(30455): make ok session=0x2a9a19d848 server=(nil) detect 10
913-wad_http_stream_get_line(976): http stream no line br_len = 225 i = 38 state 7
913-wad_http_request_reader_run(100): http reader 0x7fbffffab0 begin state=2
913-wad_http_request_reader_run(305): HTTP request method=4/7 version=3/8/0 uri=19/0 
913-bypass req(0x2a9a1ab508) caller(wad_http_init_req_status)@7908
913-wad_http_stream_get_line(976): http stream no line br_len = 187 i = 27 state 7
913-wad_http_stream_get_line(976): http stream no line br_len = 160 i = 30 state 7
913-wad_http_stream_get_line(976): http stream no line br_len = 130 i = 130 state 9
 [0x2a9a1ab508] Received request from client: 10.218.5.195:49582etc...

In the data capture above, it can be seen that the WAD PID (here 913) is now displayed in front of each method call.

In large environments, WAD debug can show a lot of information for a lot of different connections, making finding relevant information difficult. The debug output can be filtered with this option:

diagnose wad filter ?

list                    <- Display current filter.
clear                   <- Erase current filter settings.
src                     <- Source address range to filter by.
dst                     <- Destination address range to filter by.
sport                   <- Source port range to filter by.
dport                   <- Destination port range to filter by.
vd                      <- Virtual Domain Name.
explicit-policy         <- Index of explicit-policy. -1 matches all.
firewall-policy         <- Index of firewall-policy. -1 matches all.
drop-unknown-session    <- Enable drop message unknown sessions.
negate                  <- Negate the specified filter parameter.
protocol                <- Select protocols to filter by.
process-type            <- Select process type to filter by.
process-id              <- Select process ID to filter by.
process-id-by-src       <- Process id affined with the selected source IPv4/IPv6 to filter by.

Filtering for a source IP, for example, would limit debug output to traffic/authentication, etc., to/from that IP.

Note:
Even in single VDOM environments, a VDOM needs to be specified for the WAD filter.

For example:

diagnose wad filter vd root  
diagnose wad filter firewall-policy 1

The following error will appear if only the policy is specified:

diagnose wad filter firewall-policy 1
Vdom is not set.
Command fail. Return code -160

Many filters can be used at the same time to narrow down the generated logs as much as possible. 

For example:

diagnose debug console timestamp enable 
diagnose wad debug enable category ssl 
diagnose wad debug enable level verbose 
diagnose wad debug display pid enable 
diagnose wad filter src 192.168.1.1 
diagnose wad filter dst 8.8.8.8

To check WAD debug status:

diagnose wad debug show 
Category: ssl
Level: verbose
Save debug on crash: disabled
Display: pid enabled

To check WAD debug filters: 

diagnose wad filter list 
drop unknown sessions: disabled
source ip: 1.1.1.1-1.1.1.1
dest ip: 8.8.8.8-8.8.8.8

Enable debugging with: 

diagnose debug enable 

To stop debugging: 

diagnose wad debug clear
diagnose wad filter clear
diagnose debug disable
diagnose debug reset

Note:

Starting from FortiOS v8.0.0, a new CLI WAD filter has been added:

diagnose  wad filter process-id-by-src x.x.x.x
diagnose wad filter process-id-by-source 0 <---- will clear this filter only.

The command 'diagnose wad filter process-id-by-src' will create source affinity only for the traffic from the specified source to the debug worker. Using this filter, a full wad debug log is enabled for the source IP address.

Using the 'diagnose wad filter process-id-by-source 0', the original wad debug log category and level will be restored.

Related article:

Troubleshooting Tip: Example of WAD debugging for Explicit Proxy