Fortigate Internal traffic Problem

[xkalib3r]

Hi All

I have a really strange issue at my remote office. 

Basically I have a FortiWifi-40c V5.2.2 with a pretty basic configuration:

WAN1 -> Router

Internal1 -> LAN Switch

Internal 5 -> Workshop switch

We have an IPSEC tunnel up to our Head Office.

The issues we are having is that when I plug a device into the LAN (My laptop for example), I fail to get any internet breakout. After investigating, I found that the machine kept getting and IP from our local windows DHCP server, but then it would loose the IP and try and renew again (This will carry on in a constant loop). If i get on to the windows server, the DHCP adress leases get full of "BAD_Request" objects. Looking through the event logs on the windows server came back with no errors being logged.

Now the interesting this is... When I unplug the Fortigate, bam...I get a DHCP address. I then plug the Fortigate back in to the LAN switch and I can browse and access various resources.

Another interesting fact is that when connecting to the wireless on the Fortigate, I get an IP and can browse, access network resources etc, except for one thing... The branch printer (A small simple HP malfunction). I can ping the printer, but cannot print, scan or access it's web interface.

It's almost as if the Fortigate is killing internal traffic somehow. We have this same device and a very similar setup at some of our clients and have no issues. 

Yesterday I factoried the Fortigate and re-built the config from scratch, but still the issues persists. I'm pretty sure the issue started after the 5.2 upgrade, but I am unfortunately not 100% sure as most devices are wired and have never been disconnected and connected back to the network.

Today I had a look through the switch config and could not find any issues there either (Also very basic) - None the less, I firmwared the switch to the latest version in case.

Any assistance/guidance would be greatly appreciated. I would prefer avoiding a downgrade of the FortiOS if possible.

Regards 

[Christopher_McMullan]

Just an initial thought...does the FortiGate have a DHCP server set up on the internal interface facing the LAN switch?

I'm trying to think of where the BAD_request messages could be coming from. Is there a way you could show them via a screenshot? If the Windows logs don't yield any errors, it's not compromising the service itself. Would the messages be received by broadcast, maybe?

There are mechanisms for detecting address conflicts...I'm wondering if the FGT could be broadcasting BAD_request messages that get picked up by the Windows server.

Just some thoughts off-the-cuff as I nurse a coffee...

[xkalib3r]

Hi Christopher

No DHCP on the LAN interface of the Fortigate. There is however one running on our "Workshop" interface - Even though this should make no difference, I tried disabling, even unplugging this to ensure it was not causing any issues...No difference there.

I'm also thinking a broadcast of sorts. Very odd...

Screenshot attached.

[Dave_Hall]

xkalib3r wrote:

Screenshot attached.

[strike]Maybe I'm reading this wrong, but what's unusual is the MAC Address (Unique ID) listed for the bad addresses are not full MAC addresses -- they are missing some digits and the lease exp dates are all the same 2015-01-09 (RAS? IPSec?).  If I didn't know better, it looks like the DHCP lease table is corrupted on the Windows server.[/strike]

Edit: see this forum thread about a similar issue.

[xkalib3r]

Hi Guys

Thanks for all the feedback - Apologies for the late response, I have been out of the office.

Well... I came in this morning and everything is working again :\ the even more strange part... No changes have been by myself or my team. 

I was also reading up about DCHP databases being corrupt - I was going to look in to that this morning but them bam all is good again :-/. The thing with the DHCP DB being corrupt is odd though cause if I unplugged the FGT from the LAN, everything worked (DHCP wise of course)

[Dave_Hall]

One of the comments made in the thread I posted, was a laptop with IPV6 installed and the LAN/Wireless NICs were set in bridge mode.  If the 40C is the only source of wifi then I'm wondering if there isn't a laptop on your network causing something similar.

[Christopher_McMullan]

It looks like the 'name' could be Windows-generated. There still seem to be address conflicts, which was my hunch. Now, as to the source...

We could run packet captures to determine who is generating the conflicts. Two options: 'diag sniffer packet any "port 67 or port 68" 6 0 a' would show all traffic matching the filter, but the converted output in Wireshark will obfuscate the MAC addresses. It may be better to run simultaneous sniffs on individual interfaces to preserve the Layer-2 information:

diag sniff pack Internal1 "port 67 or port 68" 6 0 a

diag sniff pack Internal5 "port 67 or port 68" 6 0 a

[Christopher_McMullan]

That could have something to do with DHCP auto-configuration, based on my reading for another internal, unrelated case.

DHCP option 116 allows the server to respond with an address of 0x00000000 if the pool is exhausted. The FortiGate's default setting is:

config system dhcp server

edit <int>

...

set auto-configuration {enable | disable} default: enable

end

Having auto-configuration enabled means the FortiGate will *not* send a null address if the client broadcast includes the option. The {enable | disable} logic seems backwards, but I have that from the bug that created the feature in 2010, so unless it's wrong...

Assuming the Windows DHCP server would respond to client DHCP option 116 broadcasts, effectively triggering the client's APIPA addressing, where the FortiGate would not (by default), that might explain why removing the FortiGate seemed to resolve the issue in your case, if there were address conflicts, exhausted pools, database corruption, etc.

Just a thought...

[xkalib3r]

Hi All

Looks like the issue is still there.

Christopher - I ran the diag commands but it did not help much. I just saw My laptop in there and or any other device that I tried to connect.

In order to rule out any machines/devices causing the issue I did the following:

1. Unplugged everything from our switch (Including the firewall)

2. Logged on to the server and cleared all DHCP addresses (Note this was also after removing and re-adding the DHCP role on the box)

3. Plugged the server in to the switch

4. Plugged the firewall in to the switch.

5. Plugged a device in to the switch (I tried a phone, printer, laptop and desktop)

None of the devices picked up an IP. I then unplugged the firewall, plugged in a single device and bam, IP was successfully obtained. 

I also completely removed the config for the workshop network, leaving only our internal interface as active. 

Something to note here as well is that we have the internal (port1) and wireless interface bridged so that laptops can obtain an IP on the same subnet. Another interesting point is that when I connect a laptop to the wireless, I am connected and working 100%.

I may try and separate the wireless and LAN interfaces later today to see if that makes any difference. If that is the issue, it seems it's a v5.2.2 bug.

Doing an arp -a on the server showed me all the MAC's associated with the bad addresses in DHCP - These MAC's were just of any device that was plugged in while the firewall was plugged in. Even when testing with only one device on the lan, the bad address arp lookup showed the single device.

[xkalib3r]

Hi All

I found the issue! Well kinda...

I decided to remove the corporate wireless SSID from the soft switch and all is now working. Devices happily get IP's again!

So now the question becomes why... This has always worked and is a very standard config across all of the smaller devices that we manage :\