Description
This article describes the basic troubleshooting steps for an explicit proxy in FortiGate.
Scope
FortiGate.
Solution
The above is the logical topology used for this article.
Refer to the article below to understand the flow for reference:
Troubleshooting Tip: Example of WAD debugging for explicit proxy.
Use the following steps to troubleshoot the explicit proxy:
Step 1: Check the explicit proxy configuration.
Confirm the interface used for explicit proxy and then verify the interface:
show system interface lan
config system interface
edit "lan"
set vdom "root"
set ip 10.14.4.174 255.255.240.0
set allowaccess ping https ssh fgfm fabric
set type hard-switch
set explicit-web-proxy enable
set role lan
set snmp-index 21
next
end
Next, verify the explicit proxy port on which the firewall is listening:
show web-proxy explicit
config web-proxy explicit
set status enable
set http-incoming-port 8080
set https-incoming-port 8080
end
As per the above configuration, listening is being done on the LAN interface IP: 10.14.4.174 with port 8080.
This can also be verified in the GUI under Network -> Explicit Proxy.
It is necessary to verify the policy. In the GUI, navigate to Policy & Objects -> Proxy Policy.
In the CLI:
show firewall proxy-policy
config firewall proxy-policy
edit 1
set uuid d142c70e-e014-51ed-b1a2-d33f17e83b2e
set name "TEST"
set proxy explicit-web
set dstintf "wan1"
set srcaddr "all"
set dstaddr "all"
set service "webproxy"
set action accept
set schedule "always"
set logtraffic all
set utm-status enable
set ssl-ssh-profile "certificate-inspection"
set webfilter-profile "TEST"
next
end
Step 2: Check if the user machine is configured to forward the traffic through the proxy.
Navigate to Settings -> Network & Internet -> Proxy -> Use a proxy server.
If an automatic proxy setup is being used, additionally check to ensure 'automatically detect settings' is enabled.
Sometimes it is possible that the customer is using a PAC file, which specifically defines for which traffic only explicit proxy should be used.
Step 3:
Once ensuring the user machine is enabled with explicit proxy, the next step is to make sure that traffic of the website in the issue, is forwarded to the proxy.
One simple way to test is to go to the browser and then open Developer Tools -> Network, now it is possible to try to access the website.
Once the website access is done, it should be possible to see the requests in the developer tools, select the domain request and "Headers".
Under the Headers, it should show the remote address field which should be the proxy IP and port. If it is not the proxy IP and port, then the user machine is not forwarding the traffic to the explicit proxy for the website.
Step 4:
If the user machine is forwarding the traffic to an explicit proxy, it is necessary to verify from the firewall if receiving the traffic by taking sniffer:
di sniffer packet any 'host 10.14.4.174 and port 8080' 4 0 a
di sniffer packet any 'host 10.14.4.174 and port 8080' 4 0 a
interfaces=[any]
filters=[host 10.14.4.174 and port 8080]
2023-04-28 12:20:28.235802 lan in 10.14.3.178.64656 -> 10.14.4.174.8080: syn 3669718754
2023-04-28 12:20:28.235956 lan out 10.14.4.174.8080 -> 10.14.3.178.64656: syn 698560001 ack 3669718755
2023-04-28 12:20:28.236167 lan in 10.14.3.178.64656 -> 10.14.4.174.8080: ack 698560002
2023-04-28 12:20:28.237079 lan in 10.14.3.178.64656 -> 10.14.4.174.8080: psh 3669718755 ack 698560002
2023-04-28 12:20:28.237129 lan out 10.14.4.174.8080 -> 10.14.3.178.64656: ack 3669719239
2023-04-28 12:20:28.238984 lan out 10.14.4.174.8080 -> 10.14.3.178.64656: 698560002 ack 3669719239
2023-04-28 12:20:28.239031 lan out 10.14.4.174.8080 -> 10.14.3.178.64656: 698561462 ack 3669719239
Step 5:
The above sniffer proves the explicit proxy traffic is received from the user machine 10.14.3.178.
Now, it is necessary to make sure that the traffic is received for the website in the issue from the user's machine. For that, it is possible to take a pcap in the firewall or take a 6 0 l sniffer from the user machine IP and convert it into pcap.
In the pcap file, use the mentioned filter 'http.request.method == "CONNECT"'.
Above, it will list out all the HTTP CONNECT method packets and what is the domain for which the connection request is sent.
If checking the above, as whenever the user machine wants to connect to an explicit proxy for any website, initially it finishes TCP 3-way handshake with a firewall on the explicit proxy port and then sends an HTTP CONNECT request with the domain it wants to connect.
1 0.000000 10.14.3.178 10.14.4.174 TCP 66 51568 → 8080 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1
2 0.000185 10.14.4.174 10.14.3.178 TCP 66 8080 → 51568 [SYN, ACK] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460 SACK_PERM=1 WS=128
3 0.004159 10.14.3.178 10.14.4.174 TCP 54 51568 → 8080 [ACK] Seq=1 Ack=1 Win=262656 Len=0
4 0.006263 10.14.3.178 10.14.4.174 HTTP 276 CONNECT www.example.com:443 HTTP/1.1 <- HTTP CONNECT request for www.example.com.
If it is possible to see the CONNECT request, then make sure that the firewall is sending the 200 Connection established response.
5 0.006332 10.14.4.174 10.14.3.178 TCP 54 8080 → 51568 [ACK] Seq=1 Ack=223 Win=15744 Len=0
6 0.010479 10.14.4.174 10.14.3.178 HTTP 126 HTTP/1.1 200 Connection established <-----
'200 Connection Established' message indicates that the firewall explicit proxy was able to connect to the server to which the user is trying to connect.
Sometimes it is possible that the user will be prompted with the authentication page 'HTTP/1.1 407 Proxy authentication required'.
In this case, the user needs to authenticate and then it will show the '200 Connection established'.
The above scenario could be possible if the customer is using user groups in the explicit proxy or authentication rule.
Step 6:
Now it is necessary to check if the user is forwarding the HTTP GET request, and if receiving the response. If there is no response to the GET request, it is possible to test it by trying a ping to the domain from the firewall, this should avoid if there is any DNS resolution issue with the domain.
If it is not possible to resolve it, make sure that the DNS is configured properly on the firewall and is working fine. If the DNS resolution is working fine, make sure that the server is responding on port 443 or 80, by trying telnet from the firewall to the resolved IP.
execute ping www.example.com
PING www.example.com (10.5.20.136): 56 data bytes
64 bytes from 10.5.20.136: icmp_seq=0 ttl=255 time=0.5 ms
64 bytes from 10.5.20.136: icmp_seq=1 ttl=255 time=0.2 ms
^C
--- www.example.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.2/0.3/0.5 ms
FortiGate-81E #
FortiGate-81E #
FortiGate-81E # exe telnet 10.5.20.136 80
Trying 10.5.20.136...
Connected to 10.5.20.136.
If the above ping and telnet are working fine, it is necessary to check with the explicit proxy policy. Check if the proxy sessions are forming with the below command:
di wad filter clear
di wad filter src <x.x.x.x>
di wad session list
The above commands will show the proxy sessions successfully established from the source x.x.x.x. The sessions checked with 'di sys session list' are normal firewall policy sessions which are different from proxy sessions which are established when traffic matches explicit proxy or transparent proxy policy.
Below is the sample wad session output:
Session: explicit proxy 10.14.3.178:65525(10.5.20.174:21394)->142.250.179.74:443
id=171296463 worker=0 vd=0:0 fw-policy=1 <- Worker 0 is the worker ID handling the session, fw-policy is the explicit proxy ID which this traffic matches, and vd=0 is the VDOM ID.
duration=2 expire=3599 session-ttl=3600 <- Duration is how long the session is created, and expire is after how much time the session will be closed.
state=3 app=http sub_type=0 wan_opt_mode=0 dd_method=0
SSL enabled
to-client
SSL Port:
state=1
TCP Port:
state=2 r_blocks=1 w_blocks=0 read_blocked=0
bytes_in=1814 bytes_out=11483 shutdown=0x0
to-server
SSL Port:
state=1
TCP Port:
state=2 r_blocks=0 w_blocks=0 read_blocked=0
bytes_in=11411 bytes_out=1582 shutdown=0x0
In the above commands, 10.14.3.178 is the actual user machine IP, 10.5.20.174 is the IP of the firewall from where connection is established to the actual server, 142.250.179.74 is the actual server IP.
It is also possible to verify if there are any blocks by matching the proxy policy logs: go to GUI -> Policy & Objects -> Proxy Policy -> Select the policy intended, 'right click' and select the 'Show Matching Logs', make sure the log 'All Sessions' is enabled at least for test purpose in case the user has not enabled the option in the policy.
Step 7:
If in the above case, logs are not helpful, and the policy looks fine, capture the WAD debug output and verify. See Technical Tip: How to debug the web proxy and explicit proxy.for steps on how to capture debug output.
WAD debug:
[I][p:216][s:171296386][r:8596] wad_dump_http_request :2664 hreq=0x549a8ad0 Received request from client: 10.14.3.178:63036 <- Connect request from the user machine.
CONNECT www.example.com:443 HTTP/1.1
Host: www.example.com:443
Proxy-Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36
[I][p:216][s:171296386][r:8596] wad_http_str_canonicalize :2213 enc=0 path=/ len=1 changes=0
++ Output Omitted++
[I][p:216][s:171296386][r:8596] wad_dump_fwd_http_resp :2679 hreq=0x549a8ad0 Forward response from Internal:
HTTP/1.1 200 Connection established <-
Proxy-Agent: Fortinet-Proxy/1.0
[I][p:216] wad_accept :2383 redirect 116 accepted 10.14.3.178:51537 -> 10.14.4.174:8080 on 122 <----- Connection established on port 8080 from the client.
[V][p:216][s:171295939] wad_port_cache_make :1679 make web cache for port=0x54d0d060/(nil)
[I][p:216][s:171295939] wad_http_srv_selector_static_make :995 make static server selector.
++Output Omitted++
[I][p:216][s:171295939][r:7706] wad_dump_http_request :2664 hreq=0x549a8ad0 Received request from client: 10.14.3.178:51537 <- GET request received from the user machine, after successful Connect request response given by the firewall.
GET http://www.example.com/ HTTP/1.1 <-
Host: www.example.com <-
Proxy-Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
If-None-Match: "22322cab10abd81:0"
If-Modified-Since: Mon, 08 Aug 2022 10:21:48 GMT
[V][p:216][s:171295939][r:7706] wad_http_marker_uri :1331 scheme=http
[V][p:216][s:171295939][r:7706] wad_http_marker_uri :1288 path=/ len=1
[I][p:216][s:171295939][r:7706] wad_http_str_canonicalize :2213 enc=0 path=/ len=1 changes=0
[V][p:216][s:171295939][r:7706] wad_http_normalize_uri :2335 host_len=15 path_len=1 query_len=0
[I][p:216][s:171295939][r:7706] wad_http_req_detect_special :14924 captive_portal detected: false, preflight=(null)
[V][p:216][s:171295939][r:7706] wad_http_req_exec_act :13465 dst_addr_type=0 wc_nontp=1 sec_web=1 web_cache=0 req_bypass=0
[V][p:216][s:171295939][r:7706] wad_http_req_check_dns :66 hn=0x51e198f8 sn=(nil)
[I][p:216][s:171295939][r:7706] wad_http_dns_resolve :8010 [0x549a8ad0] DNS request name=www.example.com len=15 type/pref=0/0
[I][p:216][s:171295939][r:7706] __wad_dns_send_query :771 0:0: sending DNS request for remote peer www.example.com id=0 IPv4 <- sending DNS request for example.com.
[V][p:216][s:171295939][r:7706] wad_tcp_port_out_read_block :1005 tcp_port 0x54d0d060 fd=122 on=1 n_out_block=0~>1 in(/out)_shutdown=0/0 closed=0 state=2.
[V][p:216][s:171295939][r:7706] wad_tcp_port_transport_read_block :960 tcp_port 0x54d0d060 fd=122 on=1 n_out_block=0~>1 in(/out)_shutdown=0/0 closed=0 events=0x1.
[V][p:216][s:171295939][r:7706] wad_tcp_port_transport_read_block :974 sock 122 read_block enforced, turn off readability.
[V][p:216][s:171295939][r:7706] wad_http_msg_strm_pause :1065 strm paused, flag=0x2 is_clt=1
[V][p:216][s:171295939][r:7706] wad_http_clt_read_sync :1939 hs=0x53e3c630 pause=(1/0x2) ret=1 execute=wad_http_clt_read_req_line
[I][p:216][s:171295939][r:7706] wad_tcp_port_on_event :1963 sock 122 remove readability events=0x0.
[V][p:216][s:171295939][r:7706] wad_tcp_port_window_adjust :463 tcp_port 0x54d0d060 window-type 0 set 0 SNDBUF 327680 RCVBUF 10485760
[I][p:216] wad_dns_parse_name_resp :206 0: DNS response received for remote host www.example.com req-id=0 ipv4=1
[V][p:216] wad_dns_parse_name_resp :323 www.example.com: resp_type=1 notify=1 cdata=0 10.5.20.136
[I][p:216][s:171295939][r:7706] wad_http_dns_request_done :12438 [0x549a8ad0] DNS resolved: 10.5.20.136 <- DNS response received from the server for example.com.
++ Output Omitted++
[V][p:216][s:171295939][r:7706] wad_http_msg_strm_resume :1088 strm resumed, execute=wad_http_clt_read_req_line is_clt=1
[V][p:216][s:171295939][r:7706] wad_get_dst_intf_idx2 :220 rc = 0, new rt entry oif=5
[V][p:216][s:171295939][r:7706] wad_http_req_get_dst_intf :12170 vd=0 dst=10.5.20.136 ifidx=5
[V][p:216][s:171295939][r:7706] wad_http_req_proc_dst :12360 HTTP req=0x549a8ad0 check destination/quarantine ret=0
[V][p:216][s:171295939][r:7706]wad_http_req_check_policy :12041 start match policy vd=0(ses_ctx:x|Ph|M|Hh|C|A7|O) (10.14.3.178:51537@29->10.5.20.136:80@5) absUrl=1 <----- Proxy started matching the policy.
[I][p:216][s:171295939][r:7706] wad_fast_match_is_enable :3562 fast matching is disabled
[I][p:216][s:171295939][r:7706] wad_http_policy_match_one :352 fw_pol_id=1(pol_ctx:xh|Ad|7|=p) pflag:H|W|U|A asyn_info=1
[I][p:216][s:171295939][r:7706] wad_fw_policy_async_match :5888 pol_ctx:xh|Ad|7|=d
[I][p:216][s:171295939][r:7706] wad_http_req_policy_set :10546 match policy-id=1(pol_ctx:xh|Ad|7|=d) vd=0(ses_ctx:x|Ph|Me|Hh|C|A7|O) (10.14.3.178:51537@29 -> 10.5.20.136:80@5) <- Traffic matched the policy ID 1.
[I][p:216][s:171295939][r:7706] wad_http_req_proc_policy :10120 ses_ctx:x|Phx|Me|Hhf|C|A7|O conn_srv=0 fwd_srv=<nil>
++ Output Omitted++
[V][p:216][s:171295939][r:7706] wad_http_connect_srv :276 [0x549a8ad0] Connect to server: 10.5.20.136:80/10.5.20.136:80
[I][p:216][s:171295939][r:7706] wad_tcp_port_alloc :1464 alloc tcp_port=0x54d0e310
[V][p:216][s:171295939][r:7706] wad_tcp_port_connect_with_fd :2217 oif =5, fwd_oif=0 src_addr_unkown=1 <- Firewall did the route lookup and found the outbound interface index ID.
[I][p:216][s:171295939][r:7706] wad_tcp_port_connect_with_fd :2261 TCP port=0x54d0e310 sock=123 vrf=0 connecting 10.5.20.174:18630->10.5.20.136:80
[V][p:216][s:171295939][r:7706] wad_http_port_connect_by_task :1824 session=0x53b3bd58 client=0x54d0d060 tcp_port=0x54d0e310 ctx=0x53e39430
++ Output Omitted++
[I][p:216][s:171295939][r:7706] wad_dump_fwd_http_req :2672 hreq=0x549a8ad0 Forward request to server:
GET / HTTP/1.1
Connection: keep-alive
Host: www.example.com
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
If-None-Match: "22322cab10abd81:0"
If-Modified-Since: Mon, 08 Aug 2022 10:21:48 GMT
The above is the request forwarded to the server from the firewall explicit proxy.
[I][p:216][s:171295939] wad_dump_http_resp :2687 hreq=0x549a8ad0 Received response from server: <- Below is the response received from the server.
HTTP/1.1 200 OK
Content-Type: text/html
Last-Modified: Mon, 08 Aug 2022 10:21:48 GMT
Accept-Ranges: bytes
ETag: "22322cab10abd81:0"
Server: Microsoft-IIS/10.0
Date: Thu, 27 Apr 2023 00:12:53 GMT
Content-Length: 703
[I][p:216][s:171295939] wad_http_fwd_non_cacheable_resp :2570 resp(0x53bfdfc4) starts processing.
