Website unreachable unless specific policy is created

[bcote]

Hi,

I've come across a website that one of our department uses and since our migration to the new firewall, can't access. After a little bit of troubleshooting, I noticed that if I add a specific rule from my internal LAN to the specific website IP as a Destination, then the website is accessible. What I don't understand is that a similar rule/policy is right under it with ALL as destination and yet, the traffic doesn't go through. All other websites(so far), are reachable and I have not gotten any complaints. Why is this website different?

Can anyone explain why the ALL object doesn't seem to work for all? If I am not giving enough information, please let me know.

[rwpatterson]

What are the SERVICES associated with both policies? Are they the same? Destination is one thing, but if the all destination policy is missing a protocol, then your answer is presented. Additionally if web filtering is enabled, it may be blocked due to some other reason entirely.

[emnoc]

The cli cmd diag debug flow is your friend, I would start by analyzing that for clues.

set a filter for the website and then   remove the new policy and see what happens? What does it match? What other security profile might be taking place ?

 Also use the cli and show full for the 2 policies and diff the difference, maybe something that's not visible in the webGui is the issue.

Ken

[bcote]

Hi guys,

thanks for the quick replies. I can confirm all same services are running on both policies. That was my first guess as to why the website was unavailable, but whether I remove all of them or some on either policy, only the policy directly tied to the specific Destination is functional. I will run some debugs and see as you suggested emnoc if there is some GUI/CLI differences that could potentially cause the issue. Will follow up soon.

[emnoc]

e.g

macbook:KMgC kfelix$ ssh kfelixadm@172.16.11.142"show full  firewall policy 1222" kfelix@172.16.11.142's password: CRFWA01 $ config firewall policy     edit 1222         set name "access-to-domain-controllers"         set uuid b248e348-53469-51e6-6a99-48998d1a53bd         set srcintf "WANNTT01"         set dstintf "CORELINKE1"         set srcaddr "ALL"         set dstaddr "VIPSERVERO1W"         set rtp-nat disable         set learning-mode disable         set action accept         set status enable         set schedule "always"         set schedule-timeout disable         set service "NTP" "SSH" "ALL_ICMP"         set utm-status disable         set logtraffic utm         set logtraffic-start disable         set capture-packet disable         set auto-asic-offload enable         set wanopt disable         set webcache disable         set session-ttl 0         set vlan-cos-fwd 255         set vlan-cos-rev 255         set wccp disable         set disclaimer disable         set natip 0.0.0.0 0.0.0.0         set diffserv-forward disable         set diffserv-reverse disable         set tcp-mss-sender 0         set tcp-mss-receiver 0         set comments ''         set block-notification disable         set replacemsg-override-group ''         set srcaddr-negate disable         set dstaddr-negate disable         set service-negate disable         set timeout-send-rst disable         set captive-portal-exempt disable         set ssl-mirror disable         set scan-botnet-connections disable         set dsri disable         set delay-tcp-npu-sessoin disable         set traffic-shaper ''         set traffic-shaper-reverse ''         set per-ip-shaper ''         set nat disable         set match-vip disable     next end

[bcote]

So after copying the CLI show full of the two policies, I saw right away the difference, the reason why it isn't working is still a mystery for now.

The "test rule" which points directly to the destination also uses the Outgoing interface, while the other policy uses my NAT IP pool public IP to go out. Why can I access 90% of websites with my NAT IP Pool but not some is what I need to figure out. I noticed that the fortinet cookbook and KB technical documents give me the same results(not accessible) when going through the NAT IP.

I'll keep playing around with the policy and NAT and hopefully get debug working. For whatever reason, I cannot see any flow on the console, whether it's from SSH or directly on the Forti Console, I see nothing. I'll keep working on that too.

Thanks,

Ben

[emnoc]

Things to consider

1:  is you SNAT address blacklisted or  deny at the destination

2: maybe the SNAT address on the  other non-working  policy is not reachable ( can you ping and traceroute  to the destination using the SANT )

Ken

[bcote]

Hi Ken,

I considered #1 as the possible reason for this, but so far, when I look up the addresses, only the DUHL.Sorbs.net list comes back, but that can only affect email servers and from what I read, will not affect normal traffic. And the reason it is flagged is because some of the subnets are considered "Dynamic IP assigned" which, back in the day,Sorbs would automatically consider flagged unless the request was made to whitelist. I do want to whitelist our range so that everything is green, but I think in this case, it isn't the issue. At this point, I engaged my new provider to make sure we can do some testing on their end, but I might also get Fortinet involved to confirm all my policies, BGP configurations and such aren't the reason of the bizarre issue.

Also thought it could be an Asymmetric routing issue where for some sites the traffic would leave from one route and try to come back another.  The reason this could potentially be possible is because our Public subnets were residing on one ISP and now are on another. Even though there shouldn't be anything pointing to our old provider, I have a bad feeling there might be something there.

Ben

[emnoc]

Setup a host behind that SNAT and try a ping and traceroutes,  run a diag sniffer on the WAN(s) interfaces and validate & monitor.

To me it sounds like the SRC_NAT address is routed wrongly or blocked at the far end.  If you change the destination to some other WEBSITE, does it work?

edit:

Here's a funny story  that I wanted to share that might be happening to you.

1: A company that I worked for had a /26 routed to them

2: a single /32 in that scope was being attack by a udp flood

3: they asked for that /32 to be filter at there edge-provider ( and they did )

4: forward the clock forward  3-6 months later,  they decided to  use that /32 for a SNAT_POOL

5: it had problems, but the other 62 address in that range worked great ;)

So one more suggestion, if you change the SNAT_POOL doe that new site work? Also silly/dumb question, are you 100% sure that SRC_NAT_ADDRESS is correct ( typo/range etc..)

Ken

[bcote]

Hi Ken,

thanks again for your input in this. I am 100% the SRC_NAT_ADDRESS is correct as it is being used by all the traffic going outbound. The reason I didn't notice when we did the switch over last Friday is because of the fact that all the websites I would test worked perfectly fine. Whether I went to CNN, reddit, Google, etc... Everything would work fine. It's only when certain users went to more specific websites(related to their work)or even our Country government websites that I noticed there was something going on. For now, simply as a temporary work around on the client side, I removed the SNAT and I am simply using the ISP provided IP to go out. Everything is working well now, but it's a temporary fix. Since we are still using our old provider(that was routing the public Subnets in the past) for SIP traffic until this Friday(moving everything to the new provider), I think I will leave it as is for now. Once we do the switch over on Friday though, if the issue is still there, I'll have to work on it during the weekend. I really do feel as if it is related to the fact that our old ISP was the one routing the block and somehow there are still routes out there trying to reach our systems through that old ISP, even though they are now being advertised on our new ISP, which is what brought me to an asymmetric routing issue. 

I also did have one of my test servers behind the SNAT and under my new ISP, no go, but if I force the traffic through my old ISP(the one that still has the SIP traffic on), the server, with the same public IP, can reach the websites no problem. To me, that just reinforces the fact that it is a routing issue on the ISP end. 

and thanks for the story, made me chuckle. I did try changing the SNAT_POOL to another range(we own a full /19) and the same problem happens. 90%+ of websites work, the same ones that don't... don't. 

I'll be sure to update this post on Friday and let you know how it turned out.. or if I am still trying to figure it out :)

once more, thanks for your input

Ben