SuperUser

Solved

FortiManager deployment problems after FGT Upgrade to 7.0.14

Forum|Forum|2 years ago
February 16, 2024
17 replies
10708 views

I did the following:

- upgraded FMG to 7.0.11 while the FGT still were on 7.0.13 => everything still worked fine afterwards

- upgraded the FGT to 7.0.14 during the next night (scheduled) => since then FGT keep losing the connection to FMG when I deploy policy package or device config. Results in the deployment timing out after some time.

During a TAC session it helped to reboot FMG (and perform fsck on it with that) and then retrieving config of FGT and then deploy it. After this deploying of policy package worked fine until now.

Now just deployed the device config only on a FGT and it got disconnected from FMG again...

However they come back after some time...

Best answer by sw2090

got the new interim FOS build yesterday and it finally seems to have broought us to the right path.

This build finally outputted an additional message saying that the FMG certificate could not be re-verfified by the fgt because of the issuer. And that issue gave me the clue I needed to finally find the culprit.

It was in fact DPI in effect on the FGT to FMG policies. This was set long time ago and it never caused issues until fos 7.0.14. Since 7.0.14 this is an issue.

Once I disabled DPI on those policies everything came back up and works fine again.

Toshi_Esumi

SuperUser

I think you need to open a TT and get a TAC person to analyze what's going on when the connections are not stable. I hope that's not FMG 7.0.11's issue but might be.

Toshi

AEK

SuperUser

One of the last vulnerabilities corrected by FOS patch 7.0.14 was related to FMG communication. Follo my gaze.

AEK

sw2090Author

SuperUser

Yeah TAC ticket was already opened when i wrote this posting :)

Meanwhile had several sessions with TAC enginner and I think we might have found the culprit:

actually it seems not to be related to the security update directly but it might have indirectly caused the issue. It actually looks like that you get problems once your FGT have too many revisions in the history inside your adom. 100 seems to be a mark here that should not be exceeded.

We now limited the number of revisions to be kept in an adom and set up auto deletion of older revisions so it will not keep over 100 revisions. And since we did that it seems to work fine again.

We'll keep on monitoring the next days and TAC left the ticket still open.

That is why the update could inderectly caused the issue. If you do a firmware upgrade on a FGT that is memer of an adom in your FGt this will also trigger a retrieve config which creates a new revision and that might have striken the 100 revisions mark on our FGT :)

Just wanted to let you know here.

sw2090Author

SuperUser

hm the issue struck me again here. This morn half of my FGT were offline in FMG.

TAC told me to repair the task db which also forces a reboot of FMG. After that all FGT were back online and I could deploy one with success.

FGFMs: SSLv3/TLS read server hello

FGFMs: TLSv1.3 read encrypted extensions

FGFMs: SSLv3/TLS read server certificate request

FGFMs: SSL error: unable to get local issuer certificate

FGFMs: SSL Alert write: fatal unknown CA

FGFMs: error

FGFMs: [__get_error:846] error=1, errno=0,Success.

Also gave this to TAC who also have escalated my ticket.

Additionaly I executed a 'diag app fgfm 255' on a FGT that was offline in FMG. The Log showed there is an issue with finding a valid CA for the certificate used by FMG. This is still using the default certs here.

@sw2090 wrote:
I did the following:

- upgraded FMG to 7.0.11 while the FGT still were on 7.0.13 => everything still worked fine afterwards
- upgraded the FGT to 7.0.14 during the next night (scheduled) => since then FGT keep losing the connection to FMG when I deploy policy package or device config. Results in the deployment timing out after some time.
During a TAC session it helped to reboot FMG (and perform fsck on it with that) and then retrieving config of FGT and then deploy it. After this deploying of policy package worked fine until now.
Now just deployed the device config only on a FGT and it got disconnected from FMG again...

However they come back after some time...

sw2090Author

SuperUser

The FGFM Debug Log on the FGT also says this:

FGFMs: set_fgfm_sni SNI<support.fortinet-ca2.fortinet.com>

But the only CA on the FGT I can find that has cn=support is named FORTINET_CA_BACKUP so the SNI would be support.fortinet_ca_backup.fortinet.com I guess.

Due to this the CA is not found even though the correct ca certificate exists on the FGT.

I even checked CN and Serial and validity dates of the CAs and they are the same but the name is different between FMG and FGT.

sw2090Author

SuperUser

We updated FMG to v7.2 with TAC as they said the issue is not known in 7.2.

However it hit us again yesterday and over last night.

TAC have escalated the ticket to the developer team even.

Their last suggestion was to exclusively nail FMG to the working certificate.

I did that before but not exlusively.

To achieve this these commands can be used:

config system global

set fgfm-local-cert "Fortinet_Local2"

set fgfm-cert-exclusive enable"

end

After supplying these to our FMG all FGT came back online and I was able to deploy one that I couldn't deploy yesterday.

We'll see if that fixes it permanently...

sw2090Author

SuperUser

I think the reason is that the certificates are not new.

Same for the CAs.

Plus the culprit seems to be on FGT side though.

Maybe the certificate was not in use on FMG before 7.0.11 so nobody noticed the broken CAs on the other side.

sw2090Author

SuperUser

As said the ceritificate itself is fine on FMG side but on FGT side the CAs don't match the issuer of the certificate. And that's why the FGTs don't come back online in FMG.

sw2090Author

SuperUser

and both certificate in FMG and CAs on FGT are Fortinet Factory so cannot be modified by the user.

sw2090Author

SuperUser

We narrowed that down with TAC. It looks like if it is mainly an issue with FGT100F on 7.0.14 and FMG >= 7.0.11. Even upgrading FMG to 7.2 did not prevent it from happening. Its still escalated to the developers and pending a bugfix.

sw2090Author

SuperUser

it narrowed down to be an issue that only (at least on our side) affects the FGT100 Series. The issue never ever occured on FGT60 or FGT300 we also have.

Also the FMG Developer team has narrowed that down to be an issue on the FGT side (it is because the FGTs CAs are the culprit) and handed it over the the FGT Developer team now...

Also I got a Firmware Image from the developers that does some more debugging to get them more information. We'll see.

Will be on vacation until April 15th now but will keep you updated as Fortinet also has admitted that we are not their only customers that have this issue.

sw2090Author

SuperUser

this is still going on here. Meanwhile Fortinet TAC admittted that it is a FGT and not a FMG Issue and it affects 7.0,7.2 and also 7.4. Got this from an EMEA TAC FAZ/FMG Manager!

Our Issue also does not only affect us but also other customers.

It has escalated to Management level and also the development teams.

We're still waiting for a fix.

AEK

SuperUser

Thanks for sharing, @sw2090

Has the bug id been published?

Any advice so far for those who have FMG and want to update FG from 7.0.13 to higher patch?

AEK

sw2090Author

SuperUser

unfortunately I haven't gotten any bug id yet. But my tickets are in state "pending bugfix" and have an "E" marker for escalation.

The only adivice I coud currently give would be not to upgrade until this is fixed.

Btw. forgot to mention before: it got way worse with the last upgrade of FMG (v7.2.5). Since that my FGT remained offline and did not come back online even after downgrading FMG back to 7.2.4.

(but also affects FMG 7.0 - TAC upgraded our FMG to 7.2 some time during all this...)

Show more replies

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded