Fortiweb: configuring chunked transfer encoding for HTTP 1.1 sites

[luis_abreu]

Hello guys.

We have a couple of HTTP 1.1 web sites which sit behind our FortiWeb and we're having some issues with the chunked transfer encoding configuration. We're trying to follow the fortiweb troubleshooting guide instructions on the topicHow does Web Protection modules support Transfer.Encoding: chunked .

According to the docs, we should set chunk encoding to enable and this should be enough for ensuring that buffering won't be used for those type of requests. Unfortunately, this isn't what we're seeing after enabling this setting.

For starters, all the resources from the HTTP server are returned to the clients as chunked (even those that aren't being encoded into chunks by the http server), ie, fortiweb is transforming all http server responses into chunked transfers after enabling the previous setting (ex.: the http server will buffer js files before returning them and these requests are transformed into chunks which are then sent to the real client).

After enabling chunked transfers on FortiWeb , shouldn't FortiWeb  only apply that kind of encoding when the realt http server returns chunked encoded responses? Why is it applying it to all the responses sent back to the clien

Besides using chunked transfers for all the requests returned to the client, we've also noticed that it's only returning the first 64/65Kb for each of the resources that is used by the web page. In pratice, this means that most scripts and styles won't work properly (ex.: if page references bootstrap, it can't load the complete css or js scripts files of that library). Why is this happening?

Any ideias on how to solve this?

PS: we're running FortiWeb-1000F 7.2.10,build0409(GA),240802

Thanks.

[sjoshi]

Hi,

To configure chunked transfer encoding for HTTP 1.1 sites on FortiWeb, enable chunk encoding to prevent buffering. Ensure FortiWeb only applies chunked encoding when the real HTTP server returns chunked responses. If FortiWeb transforms all responses into chunked transfers, causing incomplete resource loading, troubleshoot by checking web protection profile configurations, verifying chunk encoding settings, and reviewing HTTP protocol constraints. If issues persist, contact Fortinet Support for further assistance. Note that configurations may vary based on your specific FortiWeb setup and environment.

[luis_abreu]

Hello.

Thanks for your answer.

We've narrowed the issue down to one of the signature collections (known exploits and trojans) applied to the Web Protection Profile that the site is using, but we're still not sure on what going on. Let me give you some more info in order to see if you can help us understand why this is happening.

Just to be sure, we've started by disabling chunked encoding. In this case, everything worked out as expected: the FortiWeb  buffered the server's response and then it returned them to the client (all resources are returned with Content-Length header set).

Then, we've re-enabled chunked transfers by running the set chunk encoding enable command. We've enabled the Web Protection Profile and HTTP Protocol Constraints we had and we've went back to the original problem: FortiWeb  returns all responses to the client as chunked and limits its size to ~64Kb.

We've went ahead and disabled the HTTP protocol constraints. Again, no go: #FortiWeb returns everything as chunked and still limits the body size of all resources to 64kb. Clearly, the issue was not related with HTTP Protocol constraints, so we've re-enabled  it and disabled the signature policy from within the Web Protection Profile (Standard Protection section -> Signatures) that is being applied to the server. After doing this, everything started working properly, ie, only chunked server responses were sent as chunked to the client. The remaining resources are buffered and returned to the client with the appropriate Content-Length header (btw, the response's body is no longer limited to ~64Kb).

Clearly the issue is being caused by one of the signature's policy or policies, so we've tried to isolate the ones that are causing problems. Our tests show that if we have the "Known exploits", "Information Disclosure" or "Trojans" signature policy are enabled in the signature collection that is applied to the Protection Web Profile, then FortiWeb  will return all the resources as chunked and limit their sizes to ~64Kb (see next image). Each of those rule groups has several individual rules and I'm really not sure on which one is breaking everything...

Any reason why this is happening? I mean, why does activating these signatures transforms all server returned responses into chunked responses? More importantly, why do they abruptly terminate the server's response and limit it to 64Kbs?

In order to workaround this issue, we've ended up setting 2 content routing policy (for testing purposes only):

* one matches the host header that identifies the site AND all urls that end up in js or css: in this case, we apply a Web Protection Profile whose signatures don't use the "Known exploits", "Information Disclosure" or "Trojans" signatures. In practice, whenever a js or css resource is returned, #FortiWeb does not translate the server's buffered response into a chunked one (and more importantly, it does limit the response's body to ~64Kb)
* another one that matches only the host header: in this case, we apply the original Web Protection Profile that uses all the signature we had initially.

Yes, we could do it the other way around too (ie, if the url *does not* match js or css, then use the full signature; otherwise, use the limited signature group).

Is there a better option for solving this problem?

Finally, one more question: according to the docs, I should be able to diagnose issues with the diagnose debug flow filter module-detail chunk-decode 7 command. We've tried to use it, but apparently it doesn't do anything. Anyone knows if it works? If so, why am I unable to see anything in the output/logs?

Thanks.