Technical Tip: Using the 'diagnose wad debug' command to troubleshoot Explicit Web Proxy related issues

[Debbie_FTNT]

Description
This article describes the '# diagnose wad debug' command and provides usage examples.

Solution
To debug traffic proxied through the FortiGate, a new WAD related diagnose command has been added to FortiOS 5.6. The main purpose of this command is to get detailed info on client/server traffic that is controlled by the WAD processes.
The '# diagnose wad debug' command has the following main options:

# diagnose wad debug ?
enable                 <----- Enable debug setting.
disable                <----- Disable debug setting.
show                       <----- Show debug setting.
clear                      <----- Clear debug setting.
display                    <----- Display setting.
save-http-req-crash        <----- Save HTTP request when WAD worker crashes.

By default, the '# diagnose wad debug' command troubleshooting options are set to the following values:

# diagnose wad debug show
Category: not set          <----- there is no category set.
Level: info                <----- debugging level is set to informational.
Display: pid disabled      <----- pid display option is disabled.

Note: In order to start capturing WAD related data, the category option must be set to something other than 'not set'.

To change the category, use '# diag wad debug enable category ?'.
The following options are available.

# diagnose wad debug show
session       <----- session.
packet        <----- packet.
dispatcher    <----- dispatcher.
http          <----- http.
cifs          <----- cifs.
mapi          <----- mapi.
socks         <----- socks.
ftp           <----- ftp.
icap          <----- icap.
ssl           <----- ssl.
webcache      <----- webcache.
bytecache     <----- byte cache
policy        <----- policy matching.
auth          <----- authentication.
scan          <----- UTM scan.
tunnel        <----- wanopt tunnel.
sys           <----- sys.
video         <----- cache video.
waf           <----- waf.
memblk             <----- memory block.
all           <----- all category.

To change the debug information level, use '# diag wad debug enable level ?'.

# diagnose wad debug enable level ?
error      <----- error.
warn       <----- warning.
info       <----- information.
verbose    <----- verbose.

To display WAD Process ID information, use the '# diag wad debug display pid ?'.

# diagnose wad debug display pid ?
enable/disable    <----- Enable/disable pid display.

To start capturing data, select a category (e.g. http) then enable debugging using '# diagnose debug enable'.

# diagnose wad debug enable category http
# diagnose debug enable

The console output will look like the following.

# wad_http_session_make(30455): make ok session=0x2a9a19d848 server=(nil) detect 10
wad_http_stream_get_line(976): http stream no line br_len = 225 i = 38 state 7
wad_http_request_reader_run(100): http reader 0x7fbffffab0 begin state=2
wad_http_request_reader_run(305): HTTP request method=4/7 version=3/8/0 uri=19/0
bypass req(0x2a9a1ab508) caller(wad_http_init_req_status)@7908
wad_http_stream_get_line(976): http stream no line br_len = 187 i = 27 state 7
wad_http_stream_get_line(976): http stream no line br_len = 160 i = 30 state 7
wad_http_stream_get_line(976): http stream no line br_len = 130 i = 130 state 9
 [0x2a9a1ab508] Received request from client: 10.218.5.195:49582etc...

To stop capturing data, use '# diagnose debug disable'.

To display the WAD Process ID that is processing the data, enable the process ID display option.

# diagnose wad debug display pid enable

To start capturing data, enable debugging using '# diagnose debug enable'.
The console output looks then like the following.

# 913-wad_http_session_make(30455): make ok session=0x2a9a19d848 server=(nil) detect 10
913-wad_http_stream_get_line(976): http stream no line br_len = 225 i = 38 state 7
913-wad_http_request_reader_run(100): http reader 0x7fbffffab0 begin state=2
913-wad_http_request_reader_run(305): HTTP request method=4/7 version=3/8/0 uri=19/0
913-bypass req(0x2a9a1ab508) caller(wad_http_init_req_status)@7908
913-wad_http_stream_get_line(976): http stream no line br_len = 187 i = 27 state 7
913-wad_http_stream_get_line(976): http stream no line br_len = 160 i = 30 state 7
913-wad_http_stream_get_line(976): http stream no line br_len = 130 i = 130 state 9
 [0x2a9a1ab508] Received request from client: 10.218.5.195:49582etc...

In the data capture above, it can be seen that the WAD PID (here 913) is now displayed in front of each method call.

In large environments, wad debug can show a lot of information for a lot of different connections, making finding relevant information difficult. Debug output can be filtered with this option:

# diagnose wad filter ?

list                    <----- Display current filter.
clear                   <----- Erase current filter settings.
src                     <----- Source address range to filter by.
dst                     <----- Destination address range to filter by.
sport                   <----- Source port range to filter by.
dport                   <----- Destination port range to filter by.
vd                      <----- Virtual Domain Name.
explicit-policy         <----- Index of explicit-policy. -1 matches all.
firewall-policy         <----- Index of firewall-policy. -1 matches all.
drop-unknown-session    <----- Enable drop message unknown sessions.
negate                  <----- Negate the specified filter parameter.
protocol                <----- Select protocols to filter by.

Filtering for a source IP, for example, would limit debug output to traffic/authentication etc to/from that IP.