Errant ARP Replies from interface with no IP.

[JaysPoorChoices]

Hi all, I'm trying to troubleshoot an odd issue and am worried that I'm missing something/out to lunch. This is a much shortened version of the issue based on this figure http://www.asciidraw.com/#Draw3254957678224072795 with the VxLan turned down for testing. 

Config Notes:

[ul]

port3 is only referenced in sys interface and the software switch.

SoftSwitch1 has l2forward and intra-switch-policy implicit.

VxLan implementation only differs from the cookbook for 5.6.6's native VxLan tunneling, so the encap destination is ike.

Tested with and without l2forward.[/ul]

The Fortigate is seeing ARP requests for 10.0.0.1 at port1 and port3 as expected. Unfortunately it's also sending replies from both ports. So clients see two replies: one with the mac for the interface (port1) that actually has that IP, and one with the mac for port3 (which blackholes any subsequent ip packets). That IP is the default gateway so it's basically blackhole-roulette depending on which reply is seen first and how fast the client caches update.

There are no other issues, VxLan tunnel appears to function perfectly when enabled. Can't reproduce the errant ARP with two Hardware Switches or with a Hardware Switch and a single port interface. 

I'd normally let TAC do their thing but I have a deadline coming soon and their initial response is that it appears to be working as expected and that they're looking for a way to disable ARP replies per-interface. That... doesn't seem right. Is there something I'm missing or don't understand? An interface should not respond to ARP requests for an IP to which it isn't bound or proxy-ing in some way, right?

The reason for the hardware switch (and not just changing to a software switch that includes the VxLan IPsec interface) is partly legacy, partly because it's in production and the backup 100D's are offsite at the moment, partly because there are a lot of references to that interface so I'll need to edit the config and apply it offline during a maintenance window. I do plan on testing that as soon as I can turn prod into dev for an hour. In the meantime I would sincerely appreciate any thoughts, suggestions, or corrections.

[gangadar1234]

can you post your config here..

[JaysPoorChoices]

gangdar1234: Sorry I missed your post, I guess I misconfigured my notification settings.

Issue one was our bad. We had a leftover undocumented VIP whose interface was set to 'any'. The 4.0 legacy docs explicitly say that the Fortigate will proxy arp requests for a VIP on any interface to which it is assigned. That might be obvious after you think about it but we didn't at the time. TAC labbed it out and got back to me the next day.   Issue two was a little more insidious. After removing the VIP we had the opposite problem: The softswitch started eating ARP requests for 10.0.0.1. I ended up posting a last-minute maintenance window where I edited the softswitch out and converted the hardware switch into a software switch that included the VxLan interface. Still don't know why that worked but it did. Seems to be running well but the feedback I got was that the VxLan feature is not commonly used.   The config matched the cookbook except that we had l2forward enabled on the involved interfaces and the physical port was plugged back into the hard-switch.

[PaulRoberts]

I hate to necropost but I ran into this very thing earlier this week because of a similar network configuration (yeah, maybe don't connect two ports to the same broadcast domain without doing something to aggregate or bond them together--it's not going to work well).  Years later it's apparently somehow still a problem?  Come on, guys.

Yes, this is an unwanted behaviour, and it's not limited to just VIPs.

While it is allowed by RFC for a host to respond to ARP queries on any interface for any address it has bound to any interface, firewalls weren't exactly "a thing" at that point in the distant past, and it's basically a behaviour meant for routers.  For a firewall, it (let me go ahead and just use RFC-like language) SHOULD NOT respond to ARP queries for IP addresses not bound to that interface, because at the very least it's information leakage (which is--cough--a low-level CVE in the making).  It would allow an attacker on the network to identify what IP addresses might be on other interfaces and then, in a worst-case scenario, bypass the firewall rules meant for the network interface that they're actually on.  The real "fix" for production environments is to stop having the same broadcast domain connecting to multiple ports (why the aggregate SFP isn't going to be affected by this will be left as an exercise to the reader), but that doesn't change the fact that a skilled attacker can very likely exploit this to bypass the firewall policies.

This has been a "known" issue for firewalls for so incredibly long.  I, personally, started fixing on my deployments with a patch to the Linux kernel roughly 30 years ago.   That same functionality was incorporated directly into the Linux kernel via the /proc interface with kernel 2.2.14 (January 28th, 1999), and then made into a sysctl for Linux Kernel 2.6.4 (March 11th, 2004).  Why TAC never communicated this issue to engineering so they could know it's long past time to set net.ipv4.conf.all.arp_ignore=2 (or if they did, why the heck it's apparently not the default) is something I'm going to try to avoid thinking about.

Note: Setting that to 2 is the right thing to do by default because it makes the Linux kernel (which we absolutely know you're using--don't be coy) ignore ARP queries that don't match an IP address bound to that specific interface, and aren't of a subnet that should exist on that interface.  ...so I while I'm on interface B I can't slap an entry into my cache for the address of interface A using the MAC address for interface B as the default gateway and then appear to be coming in via interface A as far as the firewall cares, which is where the unpleasant phrase "firewall bypass" comes out to play.  (not exactly a low-level CVE)