Visitor III

Question

VPN Phase 2 stops working after Azure FortiGate-VM Live Migration

Forum|Forum|3 years ago
October 14, 2022
16 replies
4370 views

Hello,

Trying to understand what happened and how to prevent it in the future:

- Running FortiGate-VM in an Azure VM.

- This FG has a custom site-to-site IPSec tunnel to on-prem. This effectively connects the virtual data centre to the on-premises data centre. Tunnel is initiated from Azure.

- Suddenly, the tunnel no longer works. Phase 2 will not go up.

- The first sign of trouble is this:

Unavailable : Live Migration (Unplanned)

At Thursday, October 13, 2022 at 7:29:19 PM EDT, the Azure monitoring system received the following information regarding your Virtual machine:

This virtual machine was paused for 0.675000 seconds due to a memory-preserving Live Migration operation. No additional action is required from you at this time.

Recommended Steps

No action is required

- A couple of minutes after this, alerts start going off that connectivity has been lost.

- After some trouble shooting, pinging, checking routes, connectivity, rebooting, firmware upgrade, etc. it is determined that Phase 2 simply won't go up. There are timeouts and retries, but no other obvious cause. Config has not changed anywhere, everything else seems to work just fine, it's just this phase 2 that won't work.

- I decide to recreate the tunnel on the originating side, on the FG-VM. Same exact parameters as the previous one, I literally copy / paste everything.

- Voila, tunnel immediately works again.

So:

- It was not a config change issue.

- It was not an actual connectivity issue.

It appears as if that live migration of the VM broke something. My best bet is that there's some persisted entropy, encryption key, salt, or something like that, tied to the hardware or the environment. When the live migration occurred, something stopped working because the environment changed. On physical platforms, coding something that for example uses the MAC key as a "salt" isn't a big deal, as it would never change. But on a VM, it's a problem.

1) Am I right? Or not? Could there be some other explanation as to why a tunnel needs to be re-created? If so what might be the reasons?

2) If I am right, it's now a bug, as this should not happen! VM's can move in all sorts of ways, regardless of the hosting platform (Azure, VSphere, etc.). Can't have a tunnel completely stop working and need to be recreated when a basic virtualization operation occurs.

Any insight would be much appreciated ...

Thanks,

J.F.

Anthony_E

Staff

Hello J.F,

Thank you for using the Community Forum.

I will seek to get you an answer or help. We will reply to this thread with an update as soon as possible.

Regards,

Best Regards

Anthony_E

Staff

Hello J-F,

I have found this big guide:

https://fortinetweb.s3.amazonaws.com/docs.fortinet.com/v2/attachments/df509335-8675-11eb-9995-00505692583a/FortiOS_7.0_Azure_Administration_Guide.pdf

You have some topics concerning migration.

Could you please tell me if it helps?

If not, I will continue to find a solution.

Regards,

Best Regards

jdoyonAuthor

Visitor III

I had looked at that ... the word "migration" does not feature anywhere in the document, that I could find (i.e. CTRL-F "migration" yields nothing), doing a search for the word on the web version of that doc yielded nothing either. I did the same in a few other documents without any luck.

Can you clarify what topics you saw that cover Live VM Migration?

Thanks,

J.F.

Anthony_E

Staff

Hello J-F,

In page 13:

It says migrating.

Best Regards

jdoyonAuthor

Visitor III

That's migrating between licence types, completely unrelated to Live VM Migration.

I'll try and open a ticket.

Thanks,

J.F.

Anthony_E

Staff

Thank you J-F and I hope you will have an answer for your questions.

Best Regards

hermann

Explorer

Hello,

we have faced the issue recently. 10 out of 30 VPN tunnels with different companies were affected the same way. We could recover by disabling the appropriate IPv4 firewall policies and shutting down the VPN tunnel for ca. 10 min. Shorter times like 30-60 sec didn't work.

BTW: Some other destinations stopped responding to ICMP too. Looks like non-TCP traffic is affected by such outages.

Is there a solution or a better workaround for this? FortiOS 7.4.

Kind regards,

Hermann M.

hermann

Explorer

No ideas, Fortinet? We have just faced the same issue while updating from 7.4.11 to 7.4.12.

Anthony_E

Staff

Hi,

I think T-F opened a ticket and get helped there:

https://support.fortinet.com/welcome/

Best Regards

hermann

Explorer

Of course, we did open 2 of them. Unfortunately, the Fortinet support could not find the root cause and closed them.

Anthony_E

Staff

I asked chatgpt and I get this answer:

“When dealing with a situation where the VPN Phase 2 stops working after an Azure FortiGate-VM live migration, there are several troubleshooting steps you can take to resolve the issue:

Troubleshooting Steps

Check Phase 2 Configuration:
- Ensure that the Phase 2 configuration on the FortiGate matches the settings required by Azure. This includes verifying that Perfect Forward Secrecy (PFS) is unchecked or disabled on the FortiGate.
Verify Key Lifetime:
- Confirm that the key lifetime under Phase 2 on the FortiGate is set to 27000 seconds, which is the default value on Azure. This ensures synchronization between the two ends.
Review Encryption/Authentication Settings:
- Double-check the Encryption and Authentication settings configured for the Phase 2 proposals on both the FortiGate and Azure. They must match to establish a successful VPN connection.
Check Logs for Errors:
- Look for specific log messages such as 'peer SA proposal not match local policy' which can indicate mismatches in the configuration.
Network Connectivity:
- Ensure that there is no network connectivity issue between the FortiGate and Azure that might have been introduced during the live migration.
Restart VPN Services:
- Sometimes, restarting the VPN services on the FortiGate can help re-establish the connection.”

Does it help?

Best Regards

hermann

Explorer

Thanks, Antony. I can ask AI of my choice too, no big problem.

If you read the thread, you’ll see that the issue is related to Fortigate-VMs in Azure. You can see that phase2 stops working unexpectedly. Either due to live migration of the VM to a different host or due to the firmware updates. Nothing else is changed, it does not make any sense to review phase2 parameters because they are the same as before. The network connectivity in Azure is not the topic because some VPN tunnels did not suffer the firmware update and keep working, but some of them didn’t survive and their phase2 are down completely or partly. And it’s not possible to bring them up again by pushing the button in GUI.

Yes, we see errors in the logs that phase2 go down, but we do not know why.

Even some Internet-related policies that allow ICMP traffic or other types of traffic stop working. The policies are matching the traffic (the counters are increasing) and we can see the traffic with “diag sniff” passing the Fortigate. But the traffic does not leave the Fortigate and so does not arrive at the destination.

Our workaround: Change the admin state of the impacted VPN tunnels down, disable impacted IPv4 policies. Wait “long” (20-60 min) and reenable everything.

You may agree with me that it sounds like a big issue in Fortigate-VMs for Azure.

Show more replies

Troubleshooting Steps

Sign up

Login with SSO

Login to the community

Login with SSO