Fortinet-all_rules SQL-Injection-02 false positive caused an outage!

[terren]

Hi there,

We use the Fortinet-all_rules rule for our AWS WAF (via AWS Marketplace) and have used it for a long time.

Today, we experienced an unfortunate outage due to the SQL-Injection-02 rule falsely identifying legitimate traffic to a critical endpoint in our system.

This outage was not related to any activity on our part, and AWS eventually confirmed for us that other AWS customers also experienced problems due to this false positive.

Is it the case that Fortinet pushed a problematic update to this ruleset today?  The outage began for us at 2:51pm ET. Does that line up with a deploy to that ruleset?

Thanks,

Terren

[mvogiatzis]

We faced the exact same issue with Fortinet through AWS WAF rules yesterday. Did something happened?

AWS support suggested we get in touch with Fortinet directly. They cannot offer any more help.

[terren]

Hi @mvogiatzis , when did your outage occur?  Was it around 2:51pm EST? We believe Fortinet deployed an update to their ruleset at that time.

I also sent an email to awswaf@fortinet.com, which was provided to us by AWS support. I suggest you do the same!

I did get a reply from Faiza Khan, a tech support engineer, who said only that "This issue has been fixed by our developers."  I asked for timing of when they introduced the bug and when they fixed it, but I have not heard back.

Terren

[mvogiatzis]

Thank you Teren that's helpful. YES. It was 2:50 ET!

I will email awswaf@fortinet.com and let you know here. please also reply back once you hear more. 

[mvogiatzis]

Resolved by itself at 6:30 ET. That was  using the Frankfurt region

[terren]

Not that we had much doubt, but that is great supporting evidence in the absence of validation from Fortinet. Thanks for letting me know!

[terren]

Hi all, I did receive this RCA from Fortinet today. I can't post the whole file, but here's the relevant page:

[mvogiatzis]

Thank you Terren, much appreciated. I got the same from Faiza too. Hope it doesn't happen again!