Fortiproxy BSOD Netio.sys

[sa1999]

Hello everyone,

We installed Forticlient on multiple machines and I have recently been getting BSODs on some of them. They all points to Fortiproxy sending bad arguments/parameters to NETIO.sys. The machine here is using Windows 11 22H2, and the Forticlient version at the time was 7.0.11.

Here's the stack from the minidump (happens in process Fortiproxy) :

STACK_TEXT: 

Spoiler

fffff904`caa2ea40 fffff807`7110992b       : ffff940e`d4ba0014 fffff800`7a70a000 ffff940e`00000002 ffff940e`f4dd0320     : NETIO!StreamProcessCallout+0x273

fffff904`caa2eb70 fffff807`711089dd       : 00000000`00000014 ffff940e`f4dd0320 ffff940e`eb816d00 fffff904`caa2f220     : NETIO!ProcessCallout+0xa4b

fffff904`caa2ecf0 fffff807`711076ee       : 00000000`00000000 fffff904`caa2ef20 00000000`00000001 00000000`00000000     : NETIO!ArbitrateAndEnforce+0x59d

fffff904`caa2ee20 fffff807`7115c38e       : 00000000`00000000 fffff807`6ec1d684 ffff940e`e7008080 fffff807`7112f3be     : NETIO!KfdClassify+0x33e

fffff904`caa2f1d0 fffff807`7115be58       : 00000000`00000050 ffff940e`eb816c20 00000000`00000050 ffff940e`eb816c20     : NETIO!StreamInternalClassify+0x106

fffff904`caa2f2f0 fffff807`7112eb75       : 00000000`00000014 ffff940e`eb816b50 00000000`00000000 ffff940e`f701b010     : NETIO!StreamInject+0x25c

fffff904`caa2f3c0 fffff807`71a05ebd       : ffff940e`eb816b50 00000000`00000000 00000000`00000000 ffff940e`00010000     : NETIO!FwppStreamInject+0x135

fffff904`caa2f450 fffff800`7a6c6230       : ffff940e`d430fb50 00000000`0000177b 00000000`00000000 00000000`00000040     : fwpkclnt!FwpsStreamInjectAsync0+0xfd

fffff904`caa2f4b0 ffff940e`d430fb50       : 00000000`0000177b 00000000`00000000 00000000`00000040 ffff940e`00000122     : FortiWF2+0x6230

fffff904`caa2f4b8 00000000`0000177b       : 00000000`00000000 00000000`00000040 ffff940e`00000122 00000000`00000014     : 0xffff940e`d430fb50

fffff904`caa2f4c0 00000000`00000000       : 00000000`00000040 ffff940e`00000122 00000000`00000014 fffff904`00010000     : 0x177b

I can gladly upload my minidump file if someone needs the whole thing.

SYMBOL_NAME:  NETIO!StreamProcessCallout+273

MODULE_NAME: NETIO

IMAGE_NAME:  NETIO.SYS

IMAGE_VERSION:  10.0.22621.3495

STACK_COMMAND:  .cxr 0xfffff904caa2e020 ; kb

BUCKET_ID_FUNC_OFFSET:  273

FAILURE_BUCKET_ID:  AV_NETIO!StreamProcessCallout

OS_VERSION:  10.0.22621.2506

BUILDLAB_STR:  ni_release_svc_prod3

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

FAILURE_ID_HASH:  {fb6ed777-1b72-9796-cc6e-f136919e0f6f}

I have already seen a similar forum post here: https://community.fortinet.com/t5/Support-Forum/Netio-sys-Blue-screen-of-Death-BSOD-Multiple-machine...

In the meantime, we have updated to 7.0.12, but we are not certain if this issue has been fixed.

If anybody knows anything, or if Fortinet might take up this issue, that would be great.

Thank you very much.

[johnathan]

Looks like this is potentially a known issue being tracked under Bug ID 0984763. It is caused by an incompatibility with FortiClient and certain antivirus.
You can open a ticket with TAC who can provide you an interim FortiClient to resolve this issue.

[sa1999]

Thank you for your answer Jonathan.

We are currently not using any third party antivirus, just Windows Defender. The problem also appears to be restraint to a couple of machines only. We have a whole lot of laptops, all configured the same way, and we're not getting reports for many BSODs, which would be the case if the problem was more widespread.

Is there anything else coming to your mind? Would you like me to send you the minidump file?

As an interim solution we have fully reset the affected PCs and updated them to W11 23H2. We also updated Forticlient to 7.0.12. We'll see if the problem comes back. I'll keep you posted.

[FortiNole]

Did you ever resolve?  We are having the exact same issues.  We are getting over 1k bluescreens a month at this rate across our ORG.  Running Defender and Forti 7.2.4

[sa1999]

Sadly not. We're still investigating. Some people say it's Intel drivers, but we can't definetly find it out as the problem keeps happening completely randomly.

[FortiNole]

I will update you as we progress.  Would love any new info you may find as well.  

[FortiNole]

Checking back.  Anything new on your end?  We are testing 7.0.10 per Forti.  Microsoft is still pointing to Forti as well. 

[sa1999]

Still nothing new on our end sady. Bluescreens keep happening randomly and we couldn't use certain PCs long enough yet to make sure that the issue actually went away with the things we're trying...