FortiManager deployment problems after FGT Upgrade to 7.0.14

[sw2090]

I did the following:

- upgraded FMG to 7.0.11 while the FGT still were on 7.0.13 => everything still worked fine afterwards

- upgraded the FGT to 7.0.14 during the next night (scheduled) => since then FGT keep losing the connection to FMG when I deploy policy package or device config. Results in the deployment timing out after some time. 

During a TAC session it helped to reboot FMG (and perform fsck on it with that) and then retrieving config of FGT and then deploy it. After this deploying of policy package worked fine until now.

Now just deployed the device config only on a FGT and it got disconnected from FMG again...

However they come back after some time...

[Toshi_Esumi]

I think you need to open a TT and get a TAC person to analyze what's going on when the connections are not stable. I hope that's not FMG 7.0.11's issue but might be.

Toshi

[AEK]

One of the last vulnerabilities corrected by FOS patch 7.0.14 was related to FMG communication. Follo my gaze.

[sw2090]

Yeah TAC ticket was already opened when i wrote this posting :)

Meanwhile had several sessions with TAC enginner and I think we might have found the culprit:

actually it seems not to be related to the security update directly but it might have indirectly caused the issue. It actually looks like that you get problems once your FGT have too many revisions in the history inside your adom. 100 seems  to be a mark here that should not be exceeded.

We now limited the number of revisions to be kept in an adom and set up auto deletion of older revisions so it will not keep over 100 revisions. And since we did that it seems to work fine again. 

We'll keep on monitoring the next days and TAC left the ticket still open.

That is why the update could inderectly caused the issue. If you do a firmware upgrade  on a FGT that  is memer of an adom in your FGt  this will also trigger a retrieve config which creates a new revision and that might have striken the 100 revisions mark on our FGT :)

Just wanted to let you know here.

[sw2090]

 hm the issue struck me again here. This morn half of my FGT were offline in FMG. 

TAC told me to repair the task db which also forces a reboot of FMG. After that all FGT were back online and I could deploy one with success.

FGFMs: SSLv3/TLS read server hello

FGFMs: TLSv1.3 read encrypted extensions

FGFMs: SSLv3/TLS read server certificate request

FGFMs: SSL error: unable to get local issuer certificate

FGFMs: SSL Alert write: fatal unknown CA

FGFMs: error

FGFMs: [__get_error:846] error=1, errno=0,Success.

Also gave this to TAC who also have escalated my ticket.

Additionaly I executed a 'diag app fgfm 255' on a FGT that was offline in FMG. The Log showed there is an issue with finding a valid CA for the certificate used by FMG. This is still using the default certs here. 

---

@sw2090 wrote:

I did the following:

- upgraded FMG to 7.0.11 while the FGT still were on 7.0.13 => everything still worked fine afterwards

- upgraded the FGT to 7.0.14 during the next night (scheduled) => since then FGT keep losing the connection to FMG when I deploy policy package or device config. Results in the deployment timing out after some time. 

During a TAC session it helped to reboot FMG (and perform fsck on it with that) and then retrieving config of FGT and then deploy it. After this deploying of policy package worked fine until now.

Now just deployed the device config only on a FGT and it got disconnected from FMG again...

However they come back after some time...

---

---

@sw2090 wrote:

I did the following:

- upgraded FMG to 7.0.11 while the FGT still were on 7.0.13 => everything still worked fine afterwards

- upgraded the FGT to 7.0.14 during the next night (scheduled) => since then FGT keep losing the connection to FMG when I deploy policy package or device config. Results in the deployment timing out after some time. 

During a TAC session it helped to reboot FMG (and perform fsck on it with that) and then retrieving config of FGT and then deploy it. After this deploying of policy package worked fine until now.

Now just deployed the device config only on a FGT and it got disconnected from FMG again...

However they come back after some time...

---

---

@sw2090 wrote:

I did the following:

- upgraded FMG to 7.0.11 while the FGT still were on 7.0.13 => everything still worked fine afterwards

- upgraded the FGT to 7.0.14 during the next night (scheduled) => since then FGT keep losing the connection to FMG when I deploy policy package or device config. Results in the deployment timing out after some time. 

During a TAC session it helped to reboot FMG (and perform fsck on it with that) and then retrieving config of FGT and then deploy it. After this deploying of policy package worked fine until now.

Now just deployed the device config only on a FGT and it got disconnected from FMG again...

However they come back after some time...

---

[sw2090]

The FGFM Debug Log on the FGT also says this:

FGFMs: set_fgfm_sni SNI<support.fortinet-ca2.fortinet.com>

But the only CA on the FGT I can find that has cn=support is named FORTINET_CA_BACKUP so the SNI would be support.fortinet_ca_backup.fortinet.com I guess.

Due to this the CA is not found even though the correct ca certificate exists on the FGT.

I even checked CN and Serial and validity dates of the CAs and they are the same but the name is different between FMG and FGT.

[sw2090]

We updated FMG to v7.2 with TAC as they said the issue is not known in 7.2. 

However it hit us again yesterday and over last night.

TAC have escalated the ticket to the developer team even.

Their last suggestion was to exclusively nail FMG to the working certificate. 

I did that before but not exlusively.

To achieve this these commands can be used:

config system global

  set fgfm-local-cert "Fortinet_Local2"

  set fgfm-cert-exclusive enable"

end

After supplying these to our FMG all FGT came back online and I was able to deploy one that I couldn't deploy yesterday.

We'll see if that fixes it permanently...

[AEK]

Thanks for sharing.. But I wonder how an issue linked to certificate can be intermittent.

[sw2090]

I think the reason is that the certificates are not new. 

Same for the CAs.

Plus the culprit seems to be on FGT side though.

Maybe the certificate was not in use on FMG before 7.0.11 so nobody noticed the broken CAs on the other side.

[sw2090]

As said the ceritificate itself is fine on FMG side but on FGT side the CAs don't match the issuer of the certificate. And that's why the FGTs don't come back online in FMG.