Technical note: WAD troubleshooting commands

[fgilloteau_FTNT]

Description

This article describes a list of useful commands to dump WAD proxy information.

Solution

1/  This will display the list of current authenticated users, their IP, and the time since the authentication started.

FGT04 # diagnose wad user list

ID: 2, IP: 10.0.11.142, VDOM: root

  user name   : fred@DOMAIN_TEST.LOCAL

  duration    : 124

  auth_type   : 0

  auth_method : 3

  pol_id      : 12

  g_id        : 11

  user_based  : 0

  expire      : 8

  LAN:

    bytes_in=107500 bytes_out=1169255

  WAN:

    bytes_in=799170 bytes_out=40959

2/  This will list the session in the WAD proxy. This is different from the #diag sys session list which lists the sessions in the kernel.

FGT04 # diagnose wad session  list

Session: explicit proxy 10.0.11.142:53279(10.5.20.184:11435)->172.217.18.195:443

    id=340051785 vd=0 fw-policy=12

    state=3 app=http sub_type=0 dd_mode=0 dd_method=0

    SSL enabled

    to-client

        SSL Port:

            state=3

        TCP Port:

            state=2 r_blocks=2 w_blocks=0 read_blocked=0

            bytes_in=7169 bytes_out=117563 shutdown=0x0

    to-server

        SSL Port:

            state=3

        TCP Port:

            state=2 r_blocks=0 w_blocks=0 read_blocked=0

            bytes_in=100023 bytes_out=4169 shutdown=0x0

Session: explicit proxy 10.0.11.142:53281(10.5.20.184:11438)->172.217.18.195:443

    id=340051787 vd=0 fw-policy=12

    state=3 app=http sub_type=0 dd_mode=0 dd_method=0

    SSL enabled

    to-client

        SSL Port:

            state=3

        TCP Port:

            state=2 r_blocks=2 w_blocks=0 read_blocked=0

            bytes_in=4311 bytes_out=21699 shutdown=0x0

    to-server

        SSL Port:

            state=3

        TCP Port:

            state=2 r_blocks=0 w_blocks=0 read_blocked=0

            bytes_in=4478 bytes_out=1154 shutdown=0x0

Session: explicit proxy 10.0.11.142:53282(10.5.20.184:8918)->216.58.212.174:443

    id=340051788 vd=0 fw-policy=12

    state=3 app=http sub_type=0 dd_mode=0 dd_method=0

    SSL enabled

    to-client

        SSL Port:

            state=3

        TCP Port:

            state=2 r_blocks=2 w_blocks=0 read_blocked=0

            bytes_in=4301 bytes_out=18252 shutdown=0x0

    to-server

        SSL Port:

            state=3

        TCP Port:

            state=2 r_blocks=0 w_blocks=0 read_blocked=0

            bytes_in=1011 bytes_out=1108 shutdown=0x0

Session: explicit proxy 10.0.11.142:53285(10.5.20.184:22939)->158.58.176.140:443

    id=340051791 vd=0 fw-policy=12

    state=3 app=http sub_type=0 dd_mode=0 dd_method=0

    SSL enabled

    to-client

        SSL Port:

            state=3

        TCP Port:

            state=2 r_blocks=2 w_blocks=0 read_blocked=0

            bytes_in=3472 bytes_out=20998 shutdown=0x0

    to-server

        SSL Port:

            state=3

        TCP Port:

            state=2 r_blocks=0 w_blocks=0 read_blocked=0

            bytes_in=4070 bytes_out=279 shutdown=0x0

Sessions total=4

To dump WAD commands, the FortiGate first need to enable the debug otherwise the FortiGate will not see any output"

# diag debug enable

3/  This command will list all the WAD processes.

FGT04 # diagnose test application wad 1000

Process [0]: WAD manager type=manager(0) pid=392 diagnosis=no.

Process [1]: type=dispatcher(1) index=0 pid=394 state=running

              diagnosis=no debug=enable valgrind=unsupported/disabled

Process [2]: type=wanopt(2) index=0 pid=395 state=running

              diagnosis=no debug=enable valgrind=supported/disabled

Process [3]: type=worker(3) index=0 pid=396 state=running

              diagnosis=yes debug=enable valgrind=supported/disabled

Process [4]: type=worker(3) index=1 pid=397 state=running

              diagnosis=no debug=enable valgrind=supported/disabled

Process [5]: type=worker(3) index=2 pid=398 state=running

              diagnosis=no debug=enable valgrind=supported/disabled

Process [6]: type=worker(3) index=3 pid=399 state=running

              diagnosis=no debug=enable valgrind=supported/disabled

Process [7]: type=worker(3) index=4 pid=400 state=running

              diagnosis=no debug=enable valgrind=supported/disabled

Process [8]: type=worker(3) index=5 pid=401 state=running

              diagnosis=no debug=enable valgrind=supported/disabled

Process [9]: type=worker(3) index=6 pid=402 state=running

              diagnosis=no debug=enable valgrind=supported/disabled

Process [10]: type=worker(3) index=7 pid=403 state=running

              diagnosis=no debug=enable valgrind=supported/disabled

Process [11]: type=worker(3) index=8 pid=404 state=running

              diagnosis=no debug=enable valgrind=supported/disabled

Process [12]: type=worker(3) index=9 pid=405 state=running

              diagnosis=no debug=enable valgrind=supported/disabled

Process [13]: type=informer(4) index=0 pid=393 state=running

              diagnosis=no debug=enable valgrind=unsupported/disabled

The result will be different depending on the hardware platform.

4/  This command gives the "Total shared user count"

FGT04 # diagnose test application wad 155

Total shared user count:1, shared user quota:16000, form_auth_keepalive=0 active=0, user_in_list=0

    vd=root max=0 guarantee=0 used=1

The FortiGate can also see that the MAX is 16000

The number is the count of users : authenticated users + anonymous user

An authenticated user 'fred' with IP 10.0.0.1 which sends traffic to two proxy policies, one with auth, one without auth, will count as 2 "shared user count"

5/  This will list the session handled by the WAD worker 2300

FGT_PROXY # diag test application wad 2300

FGT_PROXY # diag test application wad 21

TCP stats: active=6337 accepts=0 connects=783873 accept_err=0

           connect_err=1577 bind_fails=0 make_failure=0 connected=780015

TCP port=0x7f219a18aaf0 ses_ctx=0x7f219ab39960 sock=61/61 is_conn=0 state=2

        process=0 snfbuf=327680 rcvbuf=327680

        closed(grace/out/in/sock)=0(0/0/0/0)

        10.194.86.51:62356-->10.68.76.243:8080

TCP port=0x7f219a18b9d0 ses_ctx=0x7f219ab3ade8 sock=81/81 is_conn=0 state=2

        process=0 snfbuf=327680 rcvbuf=327680

        closed(grace/out/in/sock)=0(0/0/0/0)

        10.70.218.92:59237-->10.68.76.243:8080

..

6/  This provides statistics of SSL errors

FGT_PROXY # diag test application wad 2300

FGT_PROXY # diag test application wad 23

SSL stats:

  ports

    type-0 total 0 active 0 max 0

    type-1 total 0 active 0 max 0

    type-2 total 0 active 0 max 0

    type-3 total 0 active 0 max 0

    type-4 total 0 active 0 max 0

    type-5 total 0 active 0 max 0

    type-6 total 0 active 0 max 0

    type-7 total 0 active 0 max 0

    type-8 total 0 active 0 max 0

    type-9 total 0 active 0 max 0

  to-client:

            handshakes: started 0 completed 0 abbreviated 0 renegotiated 0 renegotiations blocked 0 insecure-renegotiations blocked 0

            session states: active 0 total 0 max 0

            cipher-suite failures 0

            session ticket: offered 0 issued 0 regenerated 0 verified 0 accepted 0 malformed 0 cipher_mismatch 0 compressor_mismatch 0 version_mismatch 0 malformed_extension 0 ems_mismatch 0

  to-server:

            handshakes: started 0 completed 0 abbreviated 0

            session states: active 0 total 0 max 0

            cipher-suite failures 0

            session ticket: offered 0 issued 0 regenerated 0 verified 0 accepted 0 malformed 0 cipher_mismatch 0 compressor_mismatch 0 version_mismatch 0 malformed_extension 0 ems_mismatch 0

  ssl proxy:

            handshakes: started 0 completed 0 abbreviated 0 renegotiated 0 renegotiations blocked 0 insecure-renegotiations blocked 0

            cipher-suite failures 0

            session states: active 0 total 0 max 0

            session ticket: offered 0 issued 0 regenerated 0 verified 0 accepted 0 malformed 0 cipher_mismatch 0 compressor_mismatch 0 version_mismatch 0 malformed_extension 0 ems_mismatch 0

  1-way forged handshake 0

  internal error 0

  bad handshake length 0

  bad change cipher spec length 0

  decrption failure 0

  hash mismatch in finished rec 0

  invalid dh size 0

  pubkey too big 0

  cert auth error 0

7/  This provides statistics about DNS resolutions initiated by the WAD proxy.

FGT_PROXY # diag test application wad 2300

FGT_PROXY # diag test application wad 104

FGT_PROXY # get test wad 104

DNS Stats: n_dns_reqs=1152 n_dns_fails=13 n_dns_timeout=0 n_dns_success=1139

           n_snd_retries=0 n_snd_fails=0 n_snd_success=1152 n_dns_overflow=0

           n_build_fails=0, n_allocated_id=0, n_dns_id_full=0

8/ This provides information about WAD object memory usage.

FGT_PROXY # diag test application wad 2300

FGT_PROXY # diag test application wad 803

The following commands are debug commands to troubleshoot WAD flow output live

This will display the current filter for capture

# diag wad filter list

This will clear the filter

# diag wad filter clear

This will configure a filter on src IP for debug

# diag wad filter src x.x.x.x

This will capture all type of debug messages

# diag wad debug enable category all

This will print the highest level of debug

# diag wad debug enable level verbose

This will clear and terminate the debug properly

# diag wad debug clear