Rule blocking communication with specific netblock seems to not be working.

[gt]

The brief overview is we have someone attempting to, I assume compromise, phones in our VOIP system from an IP originating in the Russian Federation. I have attempted to block their IP, the /24 range of IPs, ports used and i'm still seeing packets on the internal interface from their IP. I cannot for the life of me figure out why this is happening and isn't being blocked, it's even using a rule configured for outbound communication to get in, according to FortiView.

So I suppose the first question is, will the packet capturer, "diag sniffer packet internal 'net 185.22.155.0/24' 6" show packets that are dropped? I typically set it up so I have one set on wan1 and a separate one set on internal to see if the packet is actually getting to internal from wan1. So if I'm seeing a packet from wan 1 with src=185.22.155.18 and dst =8.27.x.x. then at the same time one on the internal showing src=8.27.x.x and dst=192.168.x.x that, to me, means the packet is getting through. As a note I also use the gui packet capturer to do the same so I can view the .pcap since I haven't gotten the script that converts ssh output to work yet.

If there's some information I can get from the CLI that will best help diagnose this please let me know, I'm still learning all the commands. For now though, here are some images from the gui showing what I'm talking about.  This image shows in FortiView that these connections are being accepted using a rule designed for outbound communications (Source interface: Internal. Destination interface: Wan1 & 2) 

This image shows two rules, the first is set to block this ip range on outbound communications and then the other for inbound. I initially didn't specify the interfaces, but that wasn't working so I tried specifying them and it's the same situation, both rules showing 0 bytes. I also initially had the service as ALL but thought maybe it needed to specify protocols, so I did that on the inbound. If I ping the offending IP the first rule shows its blocking data, but if an internal phone tries to communicate with the offending IP, like to respond "405 Registration Method Not Allowed" since they're attempting to register with our VOIP phones using SIP, it doesn't block the communication and no bytes appear on the rule.

For a more detailed look at the rule FortiView is saying the connections are using. I have checked to ensure the offending ip is not within the desination address netblock, as that one is in the 184.x.x.x range with a /24 CIDR.

Another portion that is confusing about this, I cannot for the life of me find the offending ip on any logs within the 90E's gui or within FortiCloud's interfaces. I have ensured its set to log all, so I would think it would be showing. The catchall syslog we save locally, however, does show the hits. Every line that has their IP is showing what I've quoted below. The part that confuses me is that it's only showing the dstip as being the internal IP its communicating with rather than any lines showing the initial connection with our public ip(wan1: 8.27.x.x). I don't know if that's indicative of anything or if that's just standard.

srcip=185.22.155.18 src_port=59944 dstip=192.168.X.X dst_port=5060 proto=17 src_int="wan1" dst_int="internal" policy_id=23 profile="default" voip_proto=sip kind=register action=permit status=failed

So at this point I'm not sure what is going on and why I'm unable to block communication to/from this IP. Any assistance would be greatly appreciated. Also apologies if I blocked out too much information, I'm really not sure what should/should not be shared publicly and just decided to err on the side of caution.

[emnoc · Accepted solution]

If there's some information I can get from the CLI that will best help diagnose this please let me know,

Yes the cli cmd diag debug flow and a correct filter is what you should be looking at. This will ensure your hitting the policy in question.

Qs you need to ask

Q1?  What is  fwpolicy-id configuration

Q2?  Where is it in the sequence of "other" policies

[oheigl · Accepted solution]

In the WAN -> Internal Policy which should block the rule, can you try to set the destination address to all or the VIP (DNAT 8.27.X.X:65476->192.168.X.X:5060) which is used for the SIP gateway? It seems like the application layer on the FortiGate allows it through an expectation session, like it's explained in this KB article: http://kb.fortinet.com/kb/documentLink.do?externalID=FD36565

But please try these two things and let us know if it helps.

[tbec2017 · Accepted solution]

The traffic is passing through because of the SIP session helper or SIP ALG (most likely).  

diagnose sys sip status - use this command to display the current SIP session helper activity including information about the SIP dialogs, mappings, and other SIP session help counts

diagnose sys sip-proxy stats list - use this command to display status information about the SIP sessions being processed by the SIP ALG. 

Detailed info on how VOIP is handled by a FG: http://docs.fortinet.com/uploaded/files/3611/fortigate-sip-56.pdf

*****BEFORE PERFORMING THE STEPS IN THE LINK BELOW MAKE SURE YOU READ THE ABOVE GUIDE AS THE BELOW STEPS MAY BRING DOWN VOIP IF YOU DO NOT HAVE ALL OF THE CORRECT PORTS OPENS/NAT TRANSLATIONS SET UP*****

You can disable the SIP session helper and SIP ALG by following the steps here: https://www.vatacom.com/knowledge-base/disable-sip-alg-fortigate-firewalls/

[emnoc · Accepted solution]

If there's some information I can get from the CLI that will best help diagnose this please let me know,

Yes the cli cmd diag debug flow and a correct filter is what you should be looking at. This will ensure your hitting the policy in question.

Qs you need to ask

Q1?  What is  fwpolicy-id configuration

Q2?  Where is it in the sequence of "other" policies

[gt]

Thank you for the response, I started using diag debug flow but the traffic has mysteriously ceased. At least for now, I will continue to keep an eye on it and hopefully get more detailed information if they decide to resume.

[MikePruett]

Let us know if it pops back up and we will help you get to the bottom of it

[gt]

Well, its back, or something similar at least, they use a different sip agent and its from a different IP block although from the same host baxet.ru. They are still attempting to register a phone within our network, same one as before oddly enough, all while I have two rules blocking communication to and from that IP block on any service. These rules are at the top of the list so it should be blocking before being accepted by another rule. So far though each register attempt has made it through and the phone returned a response of "Method not allowed" which also makes it through my rules.

I did the diag debug flow, set it up as it shows below and read that I needed to enable null pings in PuTTY so the session doesn't time out and I can view the results. I set it to run overnight along with a packet sniffer on the ips they've been using, but when I came in the diag debug flow session had closed, although the packet sniffer showed several packets. Is there a way I've missed so I can make it not timeout? Ideally I would want it to run overnight when they're active.

diag debug flow trace stop
diag debug flow filter clear
diag debug reset
diag debug flow filter saddr 46.17.47.128
diag debug flow filter saddr 46.17.44.112
diag debug flow filter saddr 46.17.47.89
diag debug flow filter saddr 46.17.46.239
diag debug flow filter daddr 46.17.47.128
diag debug flow filter daddr 46.17.44.112
diag debug flow filter daddr 46.17.47.89
diag debug flow filter daddr 46.17.46.239
diag debug flow show console enable
diag debug flow show function-name enable
diag debug console timestamp enable
diag debug flow trace start 999
diag debug enable

Side question is there a way in the diag debug flow to filter by, either saddr or daddr, a netblock rather than a single ip? Or do I need to input the individual IPs, couldn't find anything on google and hitting ? didn't show it as an option.

Again thanks for any help!

[gt]

First off I discovered that I can just use diag debug flow filter addr 46.17.42.136 rather than daddr and saddr, along with neglecting the line "diag debug disable". Second I was lucky enough to catch a fair amount of traffic today which will hopefully point us in the right direction. This is an example of the communication that happens when it receives a packet from the current offending IP which is 46.17.42.136. As a note I have explicit rules set up to block communication to that IP block as well as from that IP block. I even made one blocking the internal IP from contacting that specific IP and nothing is working. This all leads me to believe that SIP is handled differently, as all packets making it through the firewall are SIP UDP packets and I can ping the offending IP and be blocked which shows in the GUI under the column "Bytes". Hopefully I've provided enough information to help as this problem is absolutely driving me crazy, and apologies if I have redacted too much information below.

Diag Debug Flow commands

diag debug disable
diag debug flow trace stop
diag debug flow filter clear
diag debug reset
diag debug flow filter addr 46.17.42.136
diag debug flow filter proto 17
diag debug flow show console enable
diag debug flow show function-name enable
diag debug console timestamp enable
diag debug flow trace start 999
diag debug enable

Sample of captured flow

2017-08-22 17:48:28 id=20085 trace_id=776 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=17, 46.17.42.136:54619->8.27.X.X:65476) from wan1. "
2017-08-22 17:48:28 id=20085 trace_id=776 func=resolve_ip_tuple_fast line=4874 msg="Find an EXP session, id-000000f0."
2017-08-22 17:48:28 id=20085 trace_id=776 func=__ip_session_run_tuple line=2810 msg="DNAT 8.27.X.X:65476->192.168.X.X:5060"
2017-08-22 17:48:28 id=20085 trace_id=776 func=vf_ip_route_input_common line=2586 msg="find a route: flag=00000000 gw-192.168.X.X via internal"
2017-08-22 17:48:28 id=20085 trace_id=776 func=av_receive line=265 msg="send to application layer"
2017-08-22 17:48:28 id=20085 trace_id=777 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=17, 46.17.42.136:54619->192.168.X.X:5060) from local. "
2017-08-22 17:48:28 id=20085 trace_id=777 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-000000f0, original direction"
2017-08-22 17:48:28 id=20085 trace_id=778 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=17, 192.168.X.X:5060->46.17.42.136:54619) from internal. "
2017-08-22 17:48:28 id=20085 trace_id=778 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-000000f0, reply direction"
2017-08-22 17:48:28 id=20085 trace_id=778 func=vf_ip_route_input_common line=2586 msg="find a route: flag=04000000 gw-8.27.X.X via wan1" <---***Note this IP showed the next hop outside of our network, not the same 8.27.X.X as our WAN1.
2017-08-22 17:48:28 id=20085 trace_id=778 func=av_receive line=265 msg="send to application layer"
2017-08-22 17:48:28 id=20085 trace_id=779 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=17, 192.168.X.X:5060->46.17.42.136:54619) from local. "
2017-08-22 17:48:28 id=20085 trace_id=779 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-000000f0, reply direction"
2017-08-22 17:48:28 id=20085 trace_id=779 func=__ip_session_run_tuple line=2796 msg="SNAT 192.168.X.X->8.27.X.X:65476"

[oheigl · Accepted solution]

In the WAN -> Internal Policy which should block the rule, can you try to set the destination address to all or the VIP (DNAT 8.27.X.X:65476->192.168.X.X:5060) which is used for the SIP gateway? It seems like the application layer on the FortiGate allows it through an expectation session, like it's explained in this KB article: http://kb.fortinet.com/kb/documentLink.do?externalID=FD36565

But please try these two things and let us know if it helps.

[gt]

I think I already have rules set up like that, if I'm understanding correctly. I made a couple because I was trying to test out how I needed to create it, I realize in theory they should be redundant. 

I have run rule set for inbound traffic, WAN1 -> internal and then the baxet.ru (webhost who these probes have originated from) ip blocks as the source, and our public IP as the destination (8.27.X.X).

I have another rule that is set to WAN1 -> internal with the same source but the destination being our internal subnet, I didn't think this needed to exist since I would assume the packet would have destination as our public IP not our internal IP but was running out of ideas. Previous to both of these rules though I did have the destination set as ALL, and eventually changed it to specific IPs. I'll add another though and set it to all to see if it has any affect. However these are set as just addresses/objects not virtual IPs. When I look at the VIP portion it appears as if I have to map a specific port to one single internal IP as it isn't letting me input a range of anything. I would have thought I could map it to our internal subnet but whenever I enter it in, the range portion is greyed out, only allowing a single IP. I then thought about doing the specific port, but this port is used legitimately by our VOIP provider and would need to go to several internal IPs (phones).

In regards to that link I will read up on it and see if that will ring any bells as to what I need to change, maybe something in regards to an expectation session that doesn't want to go away and they keep using it.

[gt]

I was looking at the expectation sessions and found this one, which is referencing the internal phone that they keep communicating with. In it I noticed it was referring to traffic shapers, specifically the shaper for our VOIP provider. Assuming that the expectation session is what is allowing these connections in, and that traffic shaper creating the expectation session. I then went to look at the shaper being used and while the destination was set to only our VOIP providers IPs, the source was set to ALL. The setup looked as if it was designed for outbound rather than inbound communication but as a test I restricted the source to our internal subnet rather than ALL hoping it couldn't be "hijacked" by other source like the 46.17.42.136 ip being used now, even though I would think the destination would have to be our VOIP Provider's IPs instead of our own here.

Expectation Session

session info: proto=17 proto_state=00 duration=668 expire=-638 timeout=0 flags=0 0000000 sockflag=00000000 sockport=5060 av_idx=0 use=2
origin-shaper=VOIP Provider prio=2 guarantee 128000Bps max 0Bps traffic 217Bps drops 0B
reply-shaper=VOIP Provider prio=2 guarantee 128000Bps max 0Bps traffic 217Bps drops 0 B
per_ip_shaper=
shaping_policy_id=2 ha_id=0 policy_dir=1 tunnel=/ vlan_cos=255/255
state=new redir local nlb os rs ha_replicate inherit_sockport complex
statistic(bytes/packets/allow_err): org=0/0/0 reply=0/0/0 tuples=2
tx speed(Bps/kbps): 0/0 rx speed(Bps/kbps): 0/0
orgin->sink: org pre->post, reply pre->post dev=7->0/26->0 gwy=0.0.0.0/0.0.0.0
hook=pre dir=org act=dnat 0.0.0.0:0->8.27.XXX.XXX:65476(192.168.X.XXX:5060)
hook=pre dir=org act=noop 0.0.0.0:0->0.0.0.0:0(0.0.0.0:0)
misc=0 policy_id=23 auth_info=0 chk_client_info=0 vd=0
serial=000000f0 tos=ff/ff app_list=0 app=0 url_cat=0
dd_type=0 dd_mode=0

Referenced Shaper Policy

X# show firewall shaping-policy
config firewall shaping-policy
    edit 2
        set service "ALL"
        set dstintf "any"
        set traffic-shaper "VOIP Provider"
        set traffic-shaper-reverse "VOIP Provider"
        set srcaddr "obj_subnet_192.168.X.X/XX"
        set dstaddr "VOIP Provider Servers"
    next
end

After I had changed the source to be our internal subnet I still saw packets and diag debug flow traces from their IP to the same internal IP that the expectation session was referencing. I'm unsure if there's a wait time for something like this to take effect or if that session needs to expire before it uses the new settings or anything, but I'm attempting to read about how it's handled.

As always I appreciate any responses that can help me figure this out, I'm very new to the field and want to get as much knowledge and experience as I can.

Thanks,

gt

[tbec2017 · Accepted solution]

The traffic is passing through because of the SIP session helper or SIP ALG (most likely).  

diagnose sys sip status - use this command to display the current SIP session helper activity including information about the SIP dialogs, mappings, and other SIP session help counts

diagnose sys sip-proxy stats list - use this command to display status information about the SIP sessions being processed by the SIP ALG. 

Detailed info on how VOIP is handled by a FG: http://docs.fortinet.com/uploaded/files/3611/fortigate-sip-56.pdf

*****BEFORE PERFORMING THE STEPS IN THE LINK BELOW MAKE SURE YOU READ THE ABOVE GUIDE AS THE BELOW STEPS MAY BRING DOWN VOIP IF YOU DO NOT HAVE ALL OF THE CORRECT PORTS OPENS/NAT TRANSLATIONS SET UP*****

You can disable the SIP session helper and SIP ALG by following the steps here: https://www.vatacom.com/knowledge-base/disable-sip-alg-fortigate-firewalls/