Skip to main content
Debbie_FTNT
Staff & Editor
Debbie_FTNT
Staff & Editor
November 28, 2018

Technical Tip: Using the 'diagnose wad debug' command to troubleshoot Explicit Web Proxy related issues

  • November 28, 2018
  • 0 replies
  • 36830 views

Description

This article describes the 'diagnose wad debug' command and provides usage examples.

Scope

FortiGate.

Solution

To debug traffic proxied through the FortiGate, a WAD-related diagnose command has been added to FortiOS v5.6. The main purpose of this command is to get detailed info on client/server traffic that is controlled by the WAD processes.


The 'diagnose wad debug' command has the following main options:

  • 'enable': Enables the debug setting.

  • 'disable': Disables the debug setting.

  • 'show': Shows the debug setting.

  • 'clear': Clears the debug setting.

  • 'display': Displays the setting.

  • 'save-http-req-crash': Save an HTTP request when a WAD worker crashes.

 

diagnose wad debug ? 
enable
disable
show
clear
display
save-http-req-crash

 

By default, the 'diagnose wad debug' command troubleshooting options are set to the following values:

  • 'Category: not set' means there is no category set.

  • 'Level: info' means the debugging level is set to informational.

  • 'Display: pid disabled' means the PID display option is disabled.

 

diagnose wad debug show
Category: not set
Level: info
Display: pid disabled

 
Note: In order to start capturing WAD-related data, the category option must be set to something other than 'not set'.

To change the category, use 'diagnose wad debug enable category ?'.

The following options are available:

  • session: Session.

  • packet: Packet.

  • dispatcher: Dispatcher.

  • http: Http.

  • http2: Http2.

  • cifs: Cifs.

  • mapi: Mapi.

  • socks: Socks.

  • ftp: FTP.

  • icap: ICAP.

  • ssl: SSL.

  • webcache: Webcache.

  • bytecache: Byte cache

  • policy: Policy matching.

  • auth: Authentication.

  • scan: UTM scan.

  • tunnel: wanopt tunnel.

  • sys: Sys.

  • video: Cache video.

  • waf: WAD.

  • memblk: Memory block.

  • all: All categories.

 

diagnose wad debug enable category
session
packet
dispatcher
http
http2
cifs
mapi
socks
ftp
icap
ssl
webcache
bytecache
policy
auth
scan
tunnel
sys
video
waf
memblk
all


More category options have been added. As of FortiOS v7.6.7, the follow category options are available. Use the CLI command in 'diagnose wad debug enable category ?' in any firmware version to verify what categories of WAD debug are available.

diagnose wad debug enable category
session            session
packet             packet
dispatcher         dispatcher
http               http
http2              http2
cifs               cifs
mapi               mapi
socks              socks
ftp                ftp
icap               icap
ssl                ssl
webcache           webcache
bytecache          byte cache
policy             policy matching
auth               authentication
scan               UTM scan
tunnel             wanopt tunnel
sys                sys
video              cache video
memblk             memory block
icbsvc             icb service
vs                 virtual server
dskd               cache disks
csvc               cache service
ssh                ssh
rpc                rpc
sftp               sftp
scp                scp
tls_fingerprint    tls fintgerprint
uih                uih
antiphish          Anti-Phishing
detail             troubleshoot detail
info               user device info
config             learn config
dns                DNS proxy
devuln             device vulnerability
casb               casb
autotest           auto test
diag               debug daemon
monitor            worker health monitor
policy_test        firewall policy test
quic               quic
quic_cc            quic congestion control and loss detetction
http3              http3
ips                ips
email              user email stats
proc               process management
tfap               ztna traffic forward proxy
svc_conn           ztna service connector
ztna_portal        ztna web portal
device             device info
ztna_rev_conn      ztna reverse connector
stream             stream
cmgr               certificate management
websocket          websocket
all                all category

 
To change the debug information level, use 'diagnose wad debug enable level ?'. The verbosity of the debug output is increasing from 'error' (least detailed) to 'verbose' (full output).

  • error: Error.

  • warn: Warning.

  • info: Information.

  • verbose: Verbose.


diagnose wad debug enable level ? 
error
warn
info
verbose

 

To display WAD Process ID information, use the 'diagnose wad debug display pid ?' command.

  • This enables/disables PID display.

 

diagnose wad debug display pid ? 
enable/disable   

 

To start capturing data, select a category (e.g., http), then enable debugging using 'diagnose debug enable'.

diagnose wad debug enable category http
diagnose debug enable

 

The console output will look like the following.

 

wad_http_session_make(30455): make ok session=0x2a9a19d848 server=(nil) detect 10
wad_http_stream_get_line(976): http stream no line br_len = 225 i = 38 state 7
wad_http_request_reader_run(100): http reader 0x7fbffffab0 begin state=2
wad_http_request_reader_run(305): HTTP request method=4/7 version=3/8/0 uri=19/0 
bypass req(0x2a9a1ab508) caller(wad_http_init_req_status)@7908
wad_http_stream_get_line(976): http stream no line br_len = 187 i = 27 state 7
wad_http_stream_get_line(976): http stream no line br_len = 160 i = 30 state 7
wad_http_stream_get_line(976): http stream no line br_len = 130 i = 130 state 9
 [0x2a9a1ab508] Received request from client: 10.218.5.195:49582etc...


To stop capturing data, use 'diagnose debug disable'.

To display the WAD Process ID that is processing the data, enable the process ID display option.

 

diagnose wad debug display pid enable

 

To start capturing data, enable debugging using 'diagnose debug enable'.

 

The console output looks like the following.

 

913-wad_http_session_make(30455): make ok session=0x2a9a19d848 server=(nil) detect 10
913-wad_http_stream_get_line(976): http stream no line br_len = 225 i = 38 state 7
913-wad_http_request_reader_run(100): http reader 0x7fbffffab0 begin state=2
913-wad_http_request_reader_run(305): HTTP request method=4/7 version=3/8/0 uri=19/0 
913-bypass req(0x2a9a1ab508) caller(wad_http_init_req_status)@7908
913-wad_http_stream_get_line(976): http stream no line br_len = 187 i = 27 state 7
913-wad_http_stream_get_line(976): http stream no line br_len = 160 i = 30 state 7
913-wad_http_stream_get_line(976): http stream no line br_len = 130 i = 130 state 9
 [0x2a9a1ab508] Received request from client: 10.218.5.195:49582etc...

 

In the data capture above, it can be seen that the WAD PID (here 913) is now displayed in front of each method call.

In large environments, WAD debug can show a lot of information for a lot of different connections, making finding relevant information difficult. The debug output can be filtered with this option:

  • list: Displays current filter.

  • clear: Erase current filter settings.

  • src: Source address range to filter by.

  • dst: Destination address range to filter by.

  • sport: Source port range to filter by.

  • dport: Destination port range to filter by.

  • vd: Virtual Domain Name.

  • explicit-policy: Index of explicit-policy. -1 matches all.

  • firewall-policy: Index of firewall-policy. -1 matches all.

  • drop-unknown-session: Enable drop message unknown sessions.

  • negate: Negate the specified filter parameter.

  • protocol: Select protocols to filter by.

  • process-type: Select process type to filter by.

  • process-id: Select process ID to filter by.

  • process-id-by-src: Process ID associated with the selected source IPv4/IPv6 to filter by.

 

diagnose wad filter ?

list
clear
src
dst 
sport
dport 
vd
explicit-policy
firewall-policy
drop-unknown-session
negate
protocol 
process-type
process-id 
process-id-by-src

 

Filtering for a source IP, for example, would limit debug output to traffic/authentication, etc., to/from that IP.

Note:
Even in single VDOM environments, a VDOM needs to be specified for the WAD filter.

For example:

diagnose wad filter vd root  
diagnose wad filter firewall-policy 1

 
The following error will appear if only the policy is specified:
 

diagnose wad filter firewall-policy 1
Vdom is not set.
Command fail. Return code -160

 

Many filters can be used at the same time to narrow down the generated logs as much as possible. 

 

For example:

 

diagnose debug console timestamp enable 
diagnose wad debug enable category ssl 
diagnose wad debug enable level verbose 
diagnose wad debug display pid enable 
diagnose wad filter src 192.168.1.1 
diagnose wad filter dst 8.8.8.8

 

To check WAD debug status:

 

diagnose wad debug show 
Category: ssl
Level: verbose
Save debug on crash: disabled
Display: pid enabled

   

To check WAD debug filters: 

 

diagnose wad filter list 
drop unknown sessions: disabled
source ip: 1.1.1.1-1.1.1.1
dest ip: 8.8.8.8-8.8.8.8

 

Enable debugging with: 

 

diagnose debug enable 

   

To stop debugging: 

 

diagnose wad debug clear
diagnose wad filter clear
diagnose debug disable
diagnose debug reset

 

Note:

Starting from FortiOS v8.0.0, a new CLI WAD filter has been added:

  • 'diagnose wad filter process-id-by-source 0' will clear this filter only.


diagnose  wad filter process-id-by-src x.x.x.x
diagnose wad filter process-id-by-source 0


The command 'diagnose wad filter process-id-by-src' will create source affinity only for the traffic from the specified source to the debug worker. Using this filter, a full wad debug log is enabled for the source IP address.

Using the 'diagnose wad filter process-id-by-source 0', the original wad debug log category and level will be restored.


Related article:

Troubleshooting Tip: Example of WAD debugging for Explicit Proxy

    Terms & ConditionsAccessibility statement