Description
This article describes a list of useful commands to dump WAD proxy information.
Scope
FortiGate.
Solution
- This will display the list of current authenticated users, their IP, and the time since the authentication started.
diagnose wad user list
ID: 2, IP: 10.0.11.142, VDOM: root
user name : fred@DOMAIN_TEST.LOCAL
duration : 124
auth_type : 0
auth_method : 3
pol_id : 12
g_id : 11
user_based : 0
expire : 8
LAN:
bytes_in=107500 bytes_out=1169255
WAN:
bytes_in=799170 bytes_out=40959
- This will list the session in the WAD proxy. This is different from the #diag sys session list which lists the sessions in the kernel.
diagnose wad session list
Session: explicit proxy 10.0.11.142:53279(10.5.20.184:11435)->172.217.18.195:443
id=340051785 vd=0 fw-policy=12
state=3 app=http sub_type=0 dd_mode=0 dd_method=0
SSL enabled
to-client
SSL Port:
state=3
TCP Port:
state=2 r_blocks=2 w_blocks=0 read_blocked=0
bytes_in=7169 bytes_out=117563 shutdown=0x0
to-server
SSL Port:
state=3
TCP Port:
state=2 r_blocks=0 w_blocks=0 read_blocked=0
bytes_in=100023 bytes_out=4169 shutdown=0x0
Session: explicit proxy 10.0.11.142:53281(10.5.20.184:11438)->172.217.18.195:443
id=340051787 vd=0 fw-policy=12
state=3 app=http sub_type=0 dd_mode=0 dd_method=0
SSL enabled
to-client
SSL Port:
state=3
TCP Port:
state=2 r_blocks=2 w_blocks=0 read_blocked=0
bytes_in=4311 bytes_out=21699 shutdown=0x0
to-server
SSL Port:
state=3
TCP Port:
state=2 r_blocks=0 w_blocks=0 read_blocked=0
bytes_in=4478 bytes_out=1154 shutdown=0x0
Session: explicit proxy 10.0.11.142:53282(10.5.20.184:8918)->216.58.212.174:443
id=340051788 vd=0 fw-policy=12
state=3 app=http sub_type=0 dd_mode=0 dd_method=0
SSL enabled
to-client
SSL Port:
state=3
TCP Port:
state=2 r_blocks=2 w_blocks=0 read_blocked=0
bytes_in=4301 bytes_out=18252 shutdown=0x0
to-server
SSL Port:
state=3
TCP Port:
state=2 r_blocks=0 w_blocks=0 read_blocked=0
bytes_in=1011 bytes_out=1108 shutdown=0x0
Session: explicit proxy 10.0.11.142:53285(10.5.20.184:22939)->158.58.176.140:443
id=340051791 vd=0 fw-policy=12
state=3 app=http sub_type=0 dd_mode=0 dd_method=0
SSL enabled
to-client
SSL Port:
state=3
TCP Port:
state=2 r_blocks=2 w_blocks=0 read_blocked=0
bytes_in=3472 bytes_out=20998 shutdown=0x0
to-server
SSL Port:
state=3
TCP Port:
state=2 r_blocks=0 w_blocks=0 read_blocked=0
bytes_in=4070 bytes_out=279 shutdown=0x0
Sessions total=4
From FortiOS v7.0.x, it is possible to see the WAD session duration:
Session: explicit proxy 10.171.4.126:51685(10.109.19.105:8524)->142.250.74.205:443
id=1764692952 worker=0 vd=0:0 fw-policy=1
duration=50 expire=3562 session-ttl=3600
state=3 app=http sub_type=0 wan_opt_mode=0 dd_method=0
SSL enabled
to-client
TCP Port:
state=2 r_blocks=1 w_blocks=0 read_blocked=0
bytes_in=1592 bytes_out=101441 shutdown=0x0
to-server
TCP Port:
state=2 r_blocks=0 w_blocks=0 read_blocked=0
bytes_in=101369 bytes_out=1377 shutdown=0x0
To dump WAD commands, the FortiGate first needs to have the debug enabled, as otherwise, the FortiGate will not see any output.
diag debug enable
- This command will list all the WAD processes.
diagnose test application wad 1000
Process [0]: WAD manager type=manager(0) pid=392 diagnosis=no.
Process [1]: type=dispatcher(1) index=0 pid=394 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [2]: type=wanopt(2) index=0 pid=395 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(3) index=0 pid=396 state=running
diagnosis=yes debug=enable valgrind=supported/disabled
Process [4]: type=worker(3) index=1 pid=397 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [5]: type=worker(3) index=2 pid=398 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [6]: type=worker(3) index=3 pid=399 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [7]: type=worker(3) index=4 pid=400 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [8]: type=worker(3) index=5 pid=401 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [9]: type=worker(3) index=6 pid=402 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [10]: type=worker(3) index=7 pid=403 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [11]: type=worker(3) index=8 pid=404 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [12]: type=worker(3) index=9 pid=405 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [13]: type=informer(4) index=0 pid=393 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
The result will be different depending on the hardware platform.
- This command gives the 'Total shared user count'.
diagnose test application wad 110
Total shared user count:1, shared user quota:16000, form_auth_keepalive=0 active=0, user_in_list=0
vd=root max=0 guarantee=0 used=1
The FortiGate can also detect that the MAX is 16,000.
The number is the count of users: authenticated users + anonymous users.
The limit is hard-coded and cannot be changed. This limit is not covered by the 'Maximum Value table'.
An authenticated user 'fred' with IP 10.0.0.1, who sends traffic to two proxy policies - one with auth, one without auth - will count as 2 'shared user count'.
- This will list the session handled by the WAD worker 2300.
diag test application wad 2300
diag test application wad 21
TCP stats: active=6337 accepts=0 connects=783873 accept_err=0
connect_err=1577 bind_fails=0 make_failure=0 connected=780015
TCP port=0x7f219a18aaf0 ses_ctx=0x7f219ab39960 sock=61/61 is_conn=0 state=2
process=0 snfbuf=327680 rcvbuf=327680
closed(grace/out/in/sock)=0(0/0/0/0)
10.194.86.51:62356-->10.68.76.243:8080
TCP port=0x7f219a18b9d0 ses_ctx=0x7f219ab3ade8 sock=81/81 is_conn=0 state=2
process=0 snfbuf=327680 rcvbuf=327680
closed(grace/out/in/sock)=0(0/0/0/0)
10.70.218.92:59237-->10.68.76.243:8080
..
- This provides statistics on SSL errors.
diag test application wad 2300
diag test application wad 23
SSL stats:
ports
type-0 total 0 active 0 max 0
type-1 total 0 active 0 max 0
type-2 total 0 active 0 max 0
type-3 total 0 active 0 max 0
type-4 total 0 active 0 max 0
type-5 total 0 active 0 max 0
type-6 total 0 active 0 max 0
type-7 total 0 active 0 max 0
type-8 total 0 active 0 max 0
type-9 total 0 active 0 max 0
to-client:
handshakes: started 0 completed 0 abbreviated 0 renegotiated 0 renegotiations blocked 0 insecure-renegotiations blocked 0
session states: active 0 total 0 max 0
cipher-suite failures 0
session ticket: offered 0 issued 0 regenerated 0 verified 0 accepted 0 malformed 0 cipher_mismatch 0 compressor_mismatch 0 version_mismatch 0 malformed_extension 0 ems_mismatch 0
to-server:
handshakes: started 0 completed 0 abbreviated 0
session states: active 0 total 0 max 0
cipher-suite failures 0
session ticket: offered 0 issued 0 regenerated 0 verified 0 accepted 0 malformed 0 cipher_mismatch 0 compressor_mismatch 0 version_mismatch 0 malformed_extension 0 ems_mismatch 0
ssl proxy:
handshakes: started 0 completed 0 abbreviated 0 renegotiated 0 renegotiations blocked 0 insecure-renegotiations blocked 0
cipher-suite failures 0
session states: active 0 total 0 max 0
session ticket: offered 0 issued 0 regenerated 0 verified 0 accepted 0 malformed 0 cipher_mismatch 0 compressor_mismatch 0 version_mismatch 0 malformed_extension 0 ems_mismatch 0
1-way forged handshake 0
internal error 0
bad handshake length 0
bad change cipher spec length 0
decrption failure 0
hash mismatch in finished rec 0
invalid dh size 0
pubkey too big 0
cert auth error 0
- This provides statistics about DNS resolutions initiated by the WAD proxy.
diag test application wad 2300
diag test application wad 104
get test wad 104
DNS Stats: n_dns_reqs=1152 n_dns_fails=13 n_dns_timeout=0 n_dns_success=1139
n_snd_retries=0 n_snd_fails=0 n_snd_success=1152 n_dns_overflow=0
n_build_fails=0, n_allocated_id=0, n_dns_id_full=0
- This provides information about WAD object memory usage.
diag test application wad 2300
diag test application wad 803
The following commands are debug commands to troubleshoot WAD flow output live.
This will display the current filter for capture:
diag wad filter list
This will clear the filter:
diag wad filter clear
This will configure a filter on src IP for debug:
diag wad filter src x.x.x.x
This will capture all types of debug messages:
diag wad debug enable category all
This will print the highest level of debug
diag wad debug enable level verbose
This will clear and terminate the debug properly
diag wad debug clear
It is also useful to run this command to see if WAD is crashing with specific code that will help TAC engineers investigate the issue further:
diag debug crashlog read
