FortiGate 60D Site-to-Site VPN loses VoIP packets

[Wuschy]

Hi all,

I'm very new to this forum so I want to say hello to everyone first and I'd be very happy if I can get some help here :)

We have 4 Sites, all connected to each other (mesh) via Site-to-Site IPsec VPN Tunnels. Everything works without problems except one thing:

Let's say site B is using VoIP phone system over site A. In site A all phones are working without interruptions, but site B faces the issue of phones go offline and then reconnecting themselves. I monitored with the sniffing option and found that VoIP packets some-when just aren't transmitted for at least 40s. Usually it's running fine, they can do phone calls etc. but then irregularly packets aren't sent anymore.

I've monitored the connection between the phone system (192.168.8.30) and one desk phone (192.168.12.50). In the attached picture you can see the packets which are transferred (left side is site A, right side is site B).

For any help to find out more about this problem I'd be very happy as this is very annoying :(

If you need more information, just let me know! The tunnels are set up to allow traffic between sites without any logging / filters or anything like this. The only thing is a traffic shaper which gives priority to the VoIP.

Thanks and best regards

Wuschy

[Wuschy]

Hi Ken,

Again many thanks for your help! I really appreciate it!

So far I face these problems with your hints:

1. I have no Cisco Switches, only HP which aren't capable of the IP SLA-feature, so I can't do these measures. But just to remind you: we have no poor voice calls! The calls itself are perfect (except when dropped ;)) But the phones on site B just disconnect sometimes!

2. The "diag sys session filter | greps bps=" gives no output so far (my VoIP Policy is #9 so I set this "diag sys session filter policy 9"

3. The ESP-Replay window is enabled but no dropped ESP packets found so far

4. Sometimes we have a bad connection surfing the web but voice calls works without any issues, so I assume the traffic shaper is working as expected... so far the fortigate doesn't show high load, but when temporary disabling Web-Filter surfing is working normaly again so I think its dependent to the fortigate load!

PCAP capturing on both ends is exactly what I did but only shows that packets are sent on one side but not received on the other so i'm really stuck here :(

Many thanks and best regards

[emnoc]

PCAP capturing on both ends is exactly what I did but only shows that packets are sent on one side but not received on the other so i'm really stuck here :(  

To mimic  CSI Miami "Follow the evidence  !".

Since you have a one-way no packets issue,  start from that side and work outwards. The question are why no packets ? Is this  SIP ( signaling ) or  Voice Path related ( RTP stream ).

Have you monitor any   SIP related  status.response from either end? Do you have any CAC limits ( Call Admission Controls ) or any  Session Expiration issues? Do you have the VoIP in the middle or does the  call peers terminate end-2-end ( 2 leg no voipPBX or legs terminates to a voicePBX ). Was this site before hand and correctly? is it a new site ?

Are all fortigate running the same OS ?

Are all phones running the same firmware?

Are calls being terminate  correcting within SIP protocols  ( e.g a BYE and CANCEL are really not the same thing )

if you subscribe to a free  SIP provider, and test from the same side that has the re-ocurring issues, do you see same or similar issues.?

And finally, most modern  SIP termination devices have  diagnostic or call details statistics that's available on the phone.  

Since the problem as you describe it sound sporadic and hit-or-miss.  I don't think it's the firewall.

It sounds like it could be CAC, SIP.KeepAlives  or something else. Could be even  SIP authentication once again  sip.status response will give you clues. If you have a software client with  diagnostic/logging , I would enabled it at the problem site and test using that client.

Since the phones are   going on and offline, have you looked at the SIP.KA values and the  fwpolicy session-ttls? Set them for a weird and long value and re-monitor.

take a look at this;

http://socpuppet.blogspot.com/2015/05/sip-registering-issues-cisco-asa.html

It's cisco ASA specific but the same concepts applies regardless of the  firewall device.

And if you have any mangles SIP  headers;

http://socpuppet.blogspot.com/2014/09/a-sip-request-header-gotchas.html

And lastly, I had a problem that I recall with lucent devices and we found out a few device upgraded  the phone itself was the culprit but as usual we ( Firewall team was initially blamed ) ;)

So don't rule out anything is all that I can add.

;)

ken

[Wuschy]

Hi Ken,

Just to give you an Update:

It is SIP / Signaling related. Never got any feedback of a broken voice call! Currently we face no disconnections, so it's hard to "monitor".

All Fortigates are running the same firmware (Fortigate 60D latest official: v5.2.9)

All (Aastra/Mitel 5370ip) are running the same firmware (on Site A as on Site B)

All tracked calls were terminating with "BYE"

Didn't ever try to subscribe another provider as we have no issue at all on side A (Side B is connected to Side A)

Didn't found diagnostics on the phones, but will check with manufacturer or just deploy a Soft-VoIP-Client which might show me more.

I really think it's the firewall because ping packets where deployed through the VPN tunnel, but SIP keep-alive packets SOMETIMES not...

For the same reason I can't imagine (ok... I can't imagine many things so this might not be meaningful ;)) that it's SIP-KA related, but will try the Session-TTL after the next time it happens

Registration itself works like a charm, but thanks for the link!

All SIP Headers I found so far are OK!

So many thanks for your patience, I'll let you know, as soon as I know more!

best regards

Mathias

[emnoc]

It is SIP / Signaling related.

As in what? registration ( if applicable ) or something else? Typically imho when it's a  firewall or upstream device it effects all  SIP calls and is easy replicated . I still believe it's a KA issue and more so if your are NAT'ing.

Since these are ipsec-tunnels have you matched the  txPacket/rxPackets between both vpn peers

e.g ( execute a diag vpn tunnel name <vpn_tunnel> at the same time.

your txp should match the rxp counters and vice-versus

Next, what value is present in the UAC invite?  and relating to  KAs? Do you have KA enabled on the devices in it's parameters/profiles? Are you using device registration? if yes, are the device loosing regsistration and failing to register? ( this would be a big problem with the d UAC thinking it's registered and  the server see it as not  btw )

A all phones are working without interruptions, but site B faces the issue of phones go offline and then reconnecting themselves.

And lastly, on  the ipsec phase2-interface what's your ipsec SA keylife?  I ran into a strange problem with a FGT to ASA ipsec-tunnel where we had  this ridiculous low ipsec-SA keylife and upon a new SA the  SIP calls would hang. We re-matched the  FGT-2-ASA with a value of 3600secs and this issue went away. You could try a rdiculous high valueto see  if the problem improves in this regard.

e.g

config vpn ipsec phase2-interface

    edit < yourtunnelname >

      set keylife-type seconds      set keylifeseconds 172800

end

execute " get vpn ipsec tunnel name  mytunnel2siteLONUK details" for details on that vpn-tunnel

[Wuschy]

Hi Ken,

Sorry for late answer, we had about two weeks no issue, but the last week we had two issues again. So it is very hard to verify, where the problem might be!

The txPacket matches the rxpacket counts so far (until we have this kind of "interruption" i think)

Where can I check the UAC value? Is this related to the firewall or to our phone system?

Not calls are effected (can be, but normally not). The main issue is, that the phone gets disconnected, as they don't receive a "still alive" request.

And these requests SOMETIMES just don't go through the VPN, even if the ping packages go through at the same time!

Site A receives 12 packets ping, 4 packets SIP/AASP and Site B receives 12 packets ping only

As already written, this happens just sometimes... before last week, it was running for two weeks without any issue... and then, we might have it ten times a day!

I see that we have 43200s for KAs, so I'll change this to 3600s by now. Will let you know, if it helped!

Thanks alot and best regards

Mathias

[MikePruett]

Silly question but has anything changed between the two weeks without issue and the weeks with issue?

[Wuschy]

Thanks for feedback and the nice question ;)

No, nothing has changed! Last thing I changed was disabling the replay window as mentioned by Ken (http://socpuppet.blogspot.ch/2015/02/esp-replay-window-enabling-disable.html)

Then I was in good hope, that it has been fixed as we had two weeks no issue anymore.

But then it start happen again. As I am the only one with access to the firewalls, I can say for sure, that nothing else has been changed since then! It's really... annoying :(

[Wuschy]

It gets even stranger! Today I noted, that not necessarily all phones are affected always. For example I'm tracking one specific phone, which had no interruption today, but my mate at Site B tells me, the phone was disconnected! Also got the info, that this isn't the first time, that one phone has been disconnected and others not.

Is there a possibility to track before the firewall? Stupid question I know, as the VPN tunnel is encrypted, but if I could see, what my firewall at Site B is receiving before it decodes the Data, than I might get more information (for example if it is provider related... but I already changed once provider at Side A so... )

Any help appreciated!

Best regards

Mathias

[Wuschy]

No more ideas at all? :(

[Toshi_Esumi]

It seems to be a very good case to open a TT and let TAC to tackle with.