Q1:So you have DHCP on ISP facing interface?
Yes. But via PPPoE so it is PPPoE that is selected on the interface, not the DHCP option. But no settings besides the PPPoE username and password are hard coded by me on the FGT.
Q2:Both interface are assigned in the same local broadcast domain and with the same netmask and layer3 next-hop?
They always had the same next-hop, but prior to the unannounced change they had different default gateways. Now they have the same default gateway also.
Q3: on the PBR and bad gateway was the gateway out of the layer3 netmask?
With PPPoE I don't see what the netmask of the default gateway is, but by the separation between the IPs I am assigned and the gateway, it is seems to me it is fairly large. In the routing table, the default gateway is /32, as are my IPs. Here is what it shows right now:
Connected xxx.28.118.230/32 0 0 0.0.0.0 ppp1
Connected xxx.28.82.108/32 0 0 0.0.0.0 ppp2
Connected xxx.28.124.253/32 0 0 0.0.0.0 ppp1
Connected xxx.28.124.253/32 0 0 0.0.0.0 ppp2
Q4: in your PBR cfg did you use gateway and output device or just defined one
I think what you're asking is if I defined a destination address/mask or not? I only defined the incoming interface, source address (i.e., specific device I wanted to use the PBR), outgoing interface and gateway address.
Q5: I know it's late know, but from a case perspective could you re-create the post ISP PBR route for let's say
one destination and run a diag debug flow but also look at the layer2 arp entry diag ip arp list
Sorry but this office is across an ocean, so it would be irresponsible to try re-create such a serious issue. If I had lost the Domain VLAN as well because of this, I probably would have been getting on a plane to sort this out because everything would have died.
I think think the bug might be if both interface where dhcp-dynamic-assigned & within the same subnet ( see Q1 ). if this happen that would not be a good thing.
I agree with this. I am sure it is a very specific set of circumstances that led to all the traffic dying as it did.