Fortigate v5.4.1 - Explicit Web Proxy - HTTP Pipelining support?

[dkornit]

Hello,

We have recently started the development of a new in-house software for our servers, and have encountered issues with Fortigate's explicit web proxy. It seems like HTTP pipelining  is not working correctly.

When our software uses a direct connection to our backend servers, the network capture is as follows:

POST /rest/metrics/record?token=455832 HTTP/1.1
Host: apps.contoso.com
Connection: Keep-Alive
Keep-Alive: timeout=30, max=1000
Content-Length: 53

{"mem": 89.0, "cpu": 0.5, "diskw": "0", "diskr": "0"}POST /rest/metrics/record?token=259016 HTTP/1.1
Host: apps.contoso.com
Connection: Keep-Alive
Keep-Alive: timeout=30, max=1000
Content-Length: 53

{"mem": 89.0, "cpu": 0.5, "diskw": "0", "diskr": "0"}POST /rest/metrics/record?token=355108 HTTP/1.1
Host: apps.contoso.com
Connection: Keep-Alive
Keep-Alive: timeout=30, max=1000
Content-Length: 53

{"mem": 89.0, "cpu": 0.5, "diskw": "0", "diskr": "0"}POST /rest/metrics/record?token=345143 HTTP/1.1
Host: apps.contoso.com
Connection: Keep-Alive
Keep-Alive: timeout=30, max=1000
Content-Length: 53

{"mem": 89.0, "cpu": 0.5, "diskw": "0", "diskr": "0"}POST /rest/metrics/record?token=138773 HTTP/1.1
Host: apps.contoso.com
Connection: Keep-Alive
Keep-Alive: timeout=30, max=1000
Content-Length: 53

{"mem": 89.0, "cpu": 0.5, "diskw": "0", "diskr": "0"}HTTP/1.1 200 OK
Date: Wed, 10 Jun 2020 10:47:25 GMT
Server: Apache/2.4.29 (Ubuntu)
Last-Modified: Sun, 29 Mar 2020 18:38:26 GMT
ETag: "1e-5a202a3da9845"
Accept-Ranges: bytes
Content-Length: 32
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: application/json

{"result":1, "project_id": 1001}
HTTP/1.1 200 OK
Date: Wed, 10 Jun 2020 10:47:25 GMT
Server: Apache/2.4.29 (Ubuntu)
Last-Modified: Sun, 29 Mar 2020 18:38:26 GMT
ETag: "1e-5a202a3da9845"
Accept-Ranges: bytes
Content-Length: 32
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Content-Type: application/json

{"result":1, "project_id": 1001}
HTTP/1.1 200 OK
Date: Wed, 10 Jun 2020 10:47:25 GMT
Server: Apache/2.4.29 (Ubuntu)
Last-Modified: Sun, 29 Mar 2020 18:38:26 GMT
ETag: "1e-5a202a3da9845"
Accept-Ranges: bytes
Content-Length: 32
Keep-Alive: timeout=5, max=98
Connection: Keep-Alive
Content-Type: application/json

{"result":1, "project_id": 1001}
HTTP/1.1 200 OK
Date: Wed, 10 Jun 2020 10:47:25 GMT
Server: Apache/2.4.29 (Ubuntu)
Last-Modified: Sun, 29 Mar 2020 18:38:26 GMT
ETag: "1e-5a202a3da9845"
Accept-Ranges: bytes
Content-Length: 32
Keep-Alive: timeout=5, max=97
Connection: Keep-Alive
Content-Type: application/json

{"result":1, "project_id": 1001}
HTTP/1.1 200 OK
Date: Wed, 10 Jun 2020 10:47:25 GMT
Server: Apache/2.4.29 (Ubuntu)
Last-Modified: Sun, 29 Mar 2020 18:38:26 GMT
ETag: "1e-5a202a3da9845"
Accept-Ranges: bytes
Content-Length: 32
Keep-Alive: timeout=5, max=96
Connection: Keep-Alive
Content-Type: application/json

{"result":1, "project_id": 1001}

There are five pipelined HTTP requests being made, and right afterwards the server replies with 5 answers.

When connection through Fortigate explicit web proxy, 5 requests are being sent but only 2 requests are actually being made to the backend, which result in 2 responses (instead of 5). Furthermode, Keep alive is not respected (proxy terminates the connection before the agent does so):

CONNECT apps.contoso.com:80 HTTP/1.1
Host: apps.contoso.com

HTTP/1.1 200 Connection Established
Proxy-Agent: Fortinet-Proxy/1.0

POST /rest/metrics/record?token=206022 HTTP/1.1
Host: apps.contoso.com
Connection: Keep-Alive
Keep-Alive: timeout=30, max=1000
Content-Length: 53

{"mem": 89.0, "cpu": 0.5, "diskw": "0", "diskr": "0"}HTTP/1.1 200 OK
Date: Tue, 09 Jun 2020 18:59:07 GMT
Server: Apache/2.4.29 (Ubuntu)
Last-Modified: Sun, 29 Mar 2020 18:38:26 GMT
ETag: "1e-5a202a3da9845"
Accept-Ranges: bytes
Content-Length: 32
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: application/json

{"result":1, "project_id": 1001}
POST /rest/metrics/record?token=125692 HTTP/1.1
Host: apps.contoso.com
Connection: Keep-Alive
Keep-Alive: timeout=30, max=1000
Content-Length: 53

{"mem": 89.0, "cpu": 0.5, "diskw": "0", "diskr": "0"}POST /rest/metrics/record?token=355680 HTTP/1.1
Host: apps.contoso.com
Connection: Keep-Alive
Keep-Alive: timeout=30, max=1000
Content-Length: 53

{"mem": 89.0, "cpu": 0.5, "diskw": "0", "diskr": "0"}POST /rest/metrics/record?token=490617 HTTP/1.1
Host: apps.contoso.com
Connection: Keep-Alive
Keep-Alive: timeout=30, max=1000
Content-Length: 53

{"mem": 89.0, "cpu": 0.5, "diskw": "0", "diskr": "0"}POST /rest/metrics/record?token=157151 HTTP/1.1
Host: apps.contoso.com
Connection: Keep-Alive
Keep-Alive: timeout=30, max=1000
Content-Length: 53

{"mem": 89.0, "cpu": 0.5, "diskw": "0", "diskr": "0"}HTTP/1.1 200 OK
Date: Tue, 09 Jun 2020 18:59:07 GMT
Server: Apache/2.4.29 (Ubuntu)
Last-Modified: Sun, 29 Mar 2020 18:38:26 GMT
ETag: "1e-5a202a3da9845"
Accept-Ranges: bytes
Content-Length: 32
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Content-Type: application/json

{"result":1, "project_id": 1001}

 So clearly this is an issue with the proxy. We have also tested this with another proxy software and it worked just fine.

Is this a configuration issue in Fortigate Explicit Web Proxy? Please help!