Fortigate WAN DHCP problem

[Oluf]

Hi.

The "new" equipment from our local ISP delivers public IP's only by DHCP. We have a strange problem that keep happening from time to time. All of a sudden the Fortigate stops getting a new DHCP lease and we loose WAN connectivity.

Troubleshooting done by the ISP: Shutting the port which the Fortigate is connected to. Put the CPE in router mode with another subnet and dhcp scope and back to bridge mode again. Nothing helps.

The ISP says they get the dhcp request, sends the offer, but it looks like the Fortigate "closes its ears" and does not get the offer, from their point of view. When rebooting the Fortigate everything comes back up as normal. The strange thing is that when this first happens, it usually happens 2-3 times in a row when the lease expires, then it can work perfectly for months. This has so far happened on remote systems with companies that can not wait for me to get out there and debug on the Fortigate side, so we have just had to have some local people go over there and power cycle the Fortigate to get internet up and running again. So i have no debug info from the Fortigate.

So far this problem has shown itself on 60D, 90D and 300D. Firmware version 5.2.3, 5.2.4 and 5.2.5. I have googled the problem and have not found anyone that has this exact same problem. The release notes of said versions and those before/after does not include any known or resolved issues related to the Fortigate as a DHCP client.

The ISP says they have had a few other customers have this problem as well, and they also had Fortigates. Could this be some kind of bug between Fortigate and the DHCP server software the ISP is running?

This happened to a 90D today running 5.2.4, i will upgrade this one to 5.2.7 and see if that makes any difference. In the meantime, does anyone have any idea what could be causing this? I know debug data from the Fortigate would help a lot but unfortunately i have none at this time.

[Jeff_S]

Oluf,

I have been experiencing a similar if not the same issue with my Fortiwifi 30D and my Cable ISP.  I was able to test with a 80C and same issue.  Used a Netgear FVS315G with no issues.

My ISP (TekSavyy) sent me a new modem (different make and model) and issue is not resolved.

I have been working with Fortinet Support and we pulled some logs that indicate the issue is Teksavvy isn't sending an ACK packet at the end of the DHCP request.  I have attached a wireshark capture conversion of the fortigate logs of the requests.

Jeff

[Oluf]

Thank you for your reply and input. The ISP say they are sending the ACK from the DHCP server, but since i have no traffic log i don't know if the ACK reaches the Fortigate. I will have to do a wireshark trace myself if i keep having this problem after upgrading to 5.2.7. The problem is it can go months between each failure, so it is a pain in the ass to troubleshoot.

[Jeff_S]

Interesting.  My issue comes back after a couple days if I reset my modem or 30D.  If I just powercycle either or toggle my wan port, the issue comes back after an hour. This is been going on for 2 months.

[Oluf]

Looks like i have come a bit closer to a solution on this matter.

I called a friend of mine that works for the ISP and got access to their lab/development environment and had a troubleshooting session with them all day. This problem turned out to be more complex than first assumed.

I will write down my findings here for future reference and maybe help others that experience the same problem in the future.

I set up one 90D (v5.4.1) and one 60D (5.2.4) on each their own internet link with the same CPE equipment that we are experiencing problems on.

After a while we find out that we have different situations that trigger the same DHCP client problem. We only got to the bottom of one of them though, but still, we are making progress.

In the past we had one scenario when all our customers with fortigate devices on DHCP assigned links went down simultaneously and did not come up again. Rebooting them fixed the problem like it always does. The ISP replaced it's core service router that night, and this caused the problem. This internet link is designed so that from the  customer's point of view, in this case the fortigate, it looks like a layer2 connection all the way to the core service router. The MAC address the fortigate sees on it's wan interface is also the core service router. We reproduced this scenario in the lab by shutting the interface on the service router and changing it's MAC address (the MAC address is virtual). The expected behavior for the fortigate is to loose internet connectivity until it takes a new DHCP lease, worst case until the lease expires and it gets a new one (the ISP uses the DHCP lease to authenticate the customer's equipment). And here is where it gets interesting. The 90D with v5.4.1 takes a new lease almost immediately after "replacing" the service router. The 60D with v5.2.4 does not get a new lease even after it expires. But when i manually do a renew from the GUI, it comes right up. I assumed that the fortigate does the same when renewing manually and the lease expires, but by looking at the debug output from the dhcpc service it looks like it does something different when renewing manually. And yes, we did this test more than one time to makes sure it wasn't incidental.

Debug output when the lease expires:

timer 0x242d070(send_request -> send_request) will expire in 3 secs timer 0x242d070(send_request -> send_request) will expire in 2 secs timer 0x242d070(send_request -> send_request) will expire in 1 secs timer 0x242d070 expired, take action Sending request! Send a packet out. add hw header dst hw addr: E0:AC:F1:0C:68:51 src hw addr: 08:5B:0E:1A:7E:00 add ip udp header dhcpcd_send_packet,270:result:590, ifinde:5 unregister timer:0x242d070 register timer func=0x20e000 arg=0x242dbf8 name=send_request -> send_request Allocate a new timer Registered timer 0x242cfd8 will expiry in 18 secs timer 0x242cfd8(send_request -> send_request) will expire in 18 secs timer 0x242cfd8(send_request -> send_request) will expire in 17 secs timer 0x242cfd8(send_request -> send_request) will expire in 16 secs timer 0x242cfd8(send_request -> send_request) will expire in 15 secs timer 0x242cfd8(send_request -> send_request) will expire in 14 secs timer 0x242cfd8(send_request -> send_request) will expire in 13 secs timer 0x242cfd8(send_request -> send_request) will expire in 12 secs timer 0x242cfd8(send_request -> send_request) will expire in 11 secs timer 0x242cfd8(send_request -> send_request) will expire in 10 secs timer 0x242cfd8(send_request -> send_request) will expire in 9 secs timer 0x242cfd8(send_request -> send_request) will expire in 8 secs timer 0x242cfd8(send_request -> send_request) will expire in 7 secs timer 0x242cfd8(send_request -> send_request) will expire in 6 secs timer 0x242cfd8(send_request -> send_request) will expire in 5 secs timer 0x242cfd8(send_request -> send_request) will expire in 4 secs timer 0x242cfd8(send_request -> send_request) will expire in 3 secs timer 0x242cfd8(send_request -> send_request) will expire in 2 secs timer 0x242cfd8(send_request -> send_request) will expire in 1 secs timer 0x242cfd8 expired, take action Sending request! Send a packet out. add hw header dst hw addr: E0:AC:F1:0C:68:51 src hw addr: 08:5B:0E:1A:7E:00 add ip udp header dhcpcd_send_packet,270:result:590, ifinde:5 unregister timer:0x242cfd8 register timer func=0x20e000 arg=0x242dbf8 name=send_request -> send_request Allocate a new timer Registered timer 0x242d070 will expiry in 14 secs timer 0x242d070(send_request -> send_request) will expire in 14 secs timer 0x242d070(send_request -> send_request) will expire in 13 secs timer 0x242d070(send_request -> send_request) will expire in 12 secs timer 0x242d070(send_request -> send_request) will expire in 11 secs timer 0x242d070(send_request -> send_request) will expire in 10 secs timer 0x242d070(send_request -> send_request) will expire in 9 secs timer 0x242d070(send_request -> send_request) will expire in 8 secs timer 0x242d070(send_request -> send_request) will expire in 7 secs timer 0x242d070(send_request -> send_request) will expire in 6 secs timer 0x242d070(send_request -> send_request) will expire in 5 secs timer 0x242d070(send_request -> send_request) will expire in 4 secs timer 0x242d070(send_request -> send_request) will expire in 3 secs timer 0x242d070(send_request -> send_request) will expire in 2 secs timer 0x242d070(send_request -> send_request) will expire in 1 secs timer 0x242d070 expired, take action Sending request! Send a packet out. add hw header dst hw addr: E0:AC:F1:0C:68:51 src hw addr: 08:5B:0E:1A:7E:00 add ip udp header dhcpcd_send_packet,270:result:590, ifinde:5 unregister timer:0x242d070 register timer func=0x20e000 arg=0x242dbf8 name=send_request -> send_request Allocate a new timer Registered timer 0x242cfd8 will expiry in 20 secs timer 0x242cfd8(send_request -> send_request) will expire in 20 secs timer 0x242cfd8(send_request -> send_request) will expire in 19 secs timer 0x242cfd8(send_request -> send_request) will expire in 18 secs timer 0x242cfd8(send_request -> send_request) will expire in 17 secs timer 0x242cfd8(send_request -> send_request) will expire in 16 secs timer 0x242cfd8(send_request -> send_request) will expire in 15 secs timer 0x242cfd8(send_request -> send_request) will expire in 14 secs timer 0x242cfd8(send_request -> send_request) will expire in 13 secs timer 0x242cfd8(send_request -> send_request) will expire in 12 secs timer 0x242cfd8(send_request -> send_request) will expire in 11 secs timer 0x242cfd8(send_request -> send_request) will expire in 10 secs

It stays in this endless loop forever.

Debug output when renewing manually from GUI:

timer 0x242cfd8(send_request -> send_request) will expire in 6 secs timer 0x242cfd8(send_request -> send_request) will expire in 5 secs timer 0x242cfd8(send_request -> send_request) will expire in 4 secs timer 0x242cfd8(send_request -> send_request) will expire in 3 secs timer 0x242cfd8(send_request -> send_request) will expire in 2 secs

(This is where the renew button is hit in the GUI) update dhcpcd unregister timer:0x242cfd8 Unregistered timer func=0x20e000 arg=0x242dbf8 name=send_request -> send_request register timer func=0x20d9d8 arg=0x242dbf8 name=intfs_changed -> state_reboot Allocate a new timer Registered timer 0x242cfd8 will expiry in 0 secs timer 0x242cfd8 expired, take action state reboot. make request make dhcp message, code=3 Insert option(255), len(0) Insert option(53), len(1) Insert max message len (1458) Insert option(57), len(2) Insert client ID Insert option(61), len(7) Insert requested address (xxxxxxxx) Insert option(50), len(4) Insert requested options Insert option(55), len(9) Insert hostname Insert option(12), len(14) Insert class ID option Insert option(60), len(13) get_dhcp_msg_len, 301 too small, extend to 548 Sending request! Send a packet out. add hw header set dst hw addr as: FF:FF:FF:FF:FF:FF src hw addr: 08:5B:0E:1A:7E:00 add ip udp header dhcpcd_send_packet,270:result:590, ifinde:5 unregister timer:0x242cfd8 register timer func=0x20e000 arg=0x242dbf8 name=send_request -> send_request Allocate a new timer Registered timer 0x242d070 will expiry in 3 secs timer 0x242d070(send_request -> send_request) will expire in 3 secs fd 15 can be read now ###############3Receive packet: len=301 del hw header ether_type:0800 hw addr from: 00:01:00:01:00:07 del ip udp header final dhcp message len:259 DHCP Message received. parse dhcp options parse dhcp option buffer (19 bytes) option[53], len:1 option[61], len:7 option[54], len:4 DHO_SUBNET_MASK option is missed DHO_BROADCAST_ADDRESS option is missed DHO_ROUTERS option is missed DHCPNAK received state init. make discover make dhcp message, code=1 Insert option(255), len(0) Insert option(53), len(1) Insert max message len (1458) Insert option(57), len(2) Insert client ID Insert option(61), len(7) Insert requested address Insert option(50), len(4) Insert requested options Insert option(55), len(9) Insert hostname Insert option(12), len(14) Insert class ID option Insert option(60), len(13) get_dhcp_msg_len, 301 too small, extend to 548 Sending discover! Send a packet out. add hw header set dst hw addr as: FF:FF:FF:FF:FF:FF src hw addr: 08:5B:0E:1A:7E:00 add ip udp header dhcpcd_send_packet,270:result:590, ifinde:5 unregister timer:0x242d070 Unregistered timer func=0x20e000 arg=0x242dbf8 name=send_request -> send_request register timer func=0x20deb8 arg=0x242dbf8 name=send_discover -> send_discover Allocate a new timer Registered timer 0x242cfd8 will expiry in 3 secs timer 0x242cfd8(send_discover -> send_discover) will expire in 3 secs fd 15 can be read now ###############3Receive packet: len=374 del hw header ether_type:0800 hw addr from: 00:01:00:01:00:07 del ip udp header final dhcp message len:332 DHCP Message received. parse dhcp options parse dhcp option buffer (92 bytes) option[53], len:1 option[54], len:4 option[51], len:4 option[1], len:4 option[3], len:4 option[6], len:8 option[15], len:7 option[28], len:4 option[12], len:14 DHCPOFFER received on wan1 client Xid:0x58946FBD in state:2, pkt Xid:0x58946FBD handle received dhcp options! lease ip:xxxxxxxx lease time: 2698, renew: 0, rebind: 0 unregister timer:0x242cfd8 Unregistered timer func=0x20deb8 arg=0x242dbf8 name=send_discover -> send_discover register timer func=0x20db80 arg=0x242dbf8 name=dhcp_offer -> state_selecting Allocate a new timer Registered timer 0x242cfd8 will expiry in 0 secs timer 0x242cfd8 expired, take action state selecting. make request make dhcp message, code=3 Insert option(255), len(0) Insert option(53), len(1) Insert max message len (1458) Insert option(57), len(2) Insert client ID Insert option(61), len(7) Insert requested address (xxxxxxxx) Insert option(50), len(4) Insert server id Insert option(54), len(4) Insert requested options Insert option(55), len(9) Insert hostname Insert option(12), len(14) Insert class ID option Insert option(60), len(13) get_dhcp_msg_len, 307 too small, extend to 548 Sending request! Send a packet out. add hw header set dst hw addr as: FF:FF:FF:FF:FF:FF src hw addr: 08:5B:0E:1A:7E:00 add ip udp header dhcpcd_send_packet,270:result:590, ifinde:5 unregister timer:0x242cfd8 register timer func=0x20e000 arg=0x242dbf8 name=send_request -> send_request Allocate a new timer Registered timer 0x242d070 will expiry in 3 secs timer 0x242d070(send_request -> send_request) will expire in 3 secs fd 15 can be read now ###############3Receive packet: len=374 del hw header ether_type:0800 hw addr from: 00:01:00:01:00:07 del ip udp header final dhcp message len:332 DHCP Message received. parse dhcp options parse dhcp option buffer (92 bytes) option[53], len:1 option[54], len:4 option[51], len:4 option[1], len:4 option[3], len:4 option[6], len:8 option[15], len:7 option[28], len:4 option[12], len:14 DHCPACK received handle received dhcp options! lease ip:xxxxxxxx lease time: 7200, renew: 0, rebind: 0 Ack: expiry 7200 secs renew: 3600 secs rebind: 5400 secs binding lease make arp check Broadcasting ARPOP_REQUEST for 217.168.x.x Sending arpcheck! Send an arp packet out. add hw header set dst hw addr as: FF:FF:FF:FF:FF:FF src hw addr: 08:5B:0E:1A:7E:00 result:60 unregister timer:0x242d070 Unregistered timer func=0x20e000 arg=0x242dbf8 name=send_request -> send_request register timer func=0x20e2e8 arg=0x242dbf8 name=send_arp_check -> send_arp_check Allocate a new timer Registered timer 0x242cfd8 will expiry in 1 secs timer 0x242cfd8(send_arp_check -> send_arp_check) will expire in 1 secs fd 15 can be read now ###############3Receive packet: len=60 del hw header ether_type:0806 hw addr from: 00:01:00:01:00:07 arp packet received, len:46 A ARP packet is received. timer 0x242cfd8 expired, take action Sending arpcheck! Send an arp packet out. add hw header set dst hw addr as: FF:FF:FF:FF:FF:FF src hw addr: 08:5B:0E:1A:7E:00 result:60 unregister timer:0x242cfd8 register timer func=0x20e2e8 arg=0x242dbf8 name=send_arp_check -> send_arp_check Allocate a new timer Registered timer 0x242d070 will expiry in 1 secs timer 0x242d070(send_arp_check -> send_arp_check) will expire in 1 secs timer 0x242d070 expired, take action Sending arpcheck! Send an arp packet out. add hw header set dst hw addr as: FF:FF:FF:FF:FF:FF src hw addr: 08:5B:0E:1A:7E:00 result:60 unregister timer:0x242d070 register timer func=0x20e2e8 arg=0x242dbf8 name=send_arp_check -> send_arp_check Allocate a new timer Registered timer 0x242cfd8 will expiry in 1 secs timer 0x242cfd8(send_arp_check -> send_arp_check) will expire in 1 secs timer 0x242cfd8 expired, take action Sending arpcheck! Send an arp packet out. add hw header set dst hw addr as: FF:FF:FF:FF:FF:FF src hw addr: 08:5B:0E:1A:7E:00 result:60 unregister timer:0x242cfd8 register timer func=0x20e2e8 arg=0x242dbf8 name=send_arp_check -> send_arp_check Allocate a new timer Registered timer 0x242d070 will expiry in 1 secs timer 0x242d070(send_arp_check -> send_arp_check) will expire in 1 secs timer 0x242d070 expired, take action Sending arpcheck! bind lease unregister timer:0x242d070 register timer func=0x20dd88 arg=0x242dbf8 name=bind_lease -> state_renewing Allocate a new timer Registered timer 0x242cfd8 will expiry in 3596 secs config interface:wan1 config interface ip:xxxxxxxx Config interface netmask: E0FFFFFF Config interface broadcast: FFFFFFFF config interface default gateway:xxxxxxxx config interface dns1:xxxxxxxx config interface dns2:xxxxxxxx make arp inform Broadcasting ARPOP_REPLY for 217.168.x.x to make it valid Send an arp packet out. add hw header set dst hw addr as: FF:FF:FF:FF:FF:FF src hw addr: 08:5B:0E:1A:7E:00 result:60 send DHCPCD_HA_SYNC_T_LEASE timer 0x242cfd8(bind_lease -> state_renewing) will expire in 3596 secs fd 15 can be read now ###############3Receive packet: len=60 del hw header ether_type:0806 hw addr from: 08:5B:0E:1A:7E:00 arp packet received, len:46 A ARP packet is received. The ARP packet is from us, ignore! timer 0x242cfd8(bind_lease -> state_renewing) will expire in 3596 secs timer 0x242cfd8(bind_lease -> state_renewing) will expire in 3595 secs timer 0x242cfd8(bind_lease -> state_renewing) will expire in 3594 secs

Conclusion:

Problem is present on v5.2.4, and most 5.2.x releases i assume, since the fortigate devices that did not come up that night the ISP replaced the service router had different releases. 5.2.3, 5.2.4 and 5.2.5 that i am 100% sure of.

Problem is not present on v5.4.1. Possibly 5.4.0 also, but this was not tested.

Now, that covers my problem when the ISP replaces the service router. But i have had this problem happen many, many times. And the ISP has only replaced the core service router once in that time period. The other times this happens it is also just that one customer, and particularly one specific. So there is also other things that trigger this problem. The only similarity i can find on the customers that have this problem frequently is that they are the ones with the most generated traffic though the firewall, and having fortigate models that is in the gray area of being too small for their use. So it could be triggered by the fortigate being overworked? This would just be guessing from my side though as i have done no debugging on those said fortigates. I also don't know if v5.4.1 will fix that, but that will be my next thing to try.

[Jeff_S]

I was on 5.4.0 when my issue started.  I went back to 5.2.3 which I believe I was on before 5.4.0 and then 5.2.7 and issue just followed.  I have now upgraded to 5.4.1 to test from my end and still having the issue.  My dhcp logs look like your second log entry.

DHCP Trace

timer 0x2a8bc08(send_request -> send_request) will expire in 4 secs timer 0x2a8bc08(send_request -> send_request) will expire in 3 secs timer 0x2a8bc08(send_request -> send_request) will expire in 2 secs timer 0x2a8bc08(send_request -> send_request) will expire in 1 secs timer 0x2a8bc08 expired, take action Sending request! Send a packet out. add hw header set dst hw addr as: FF:FF:FF:FF:FF:FF src hw addr: 08:5B:0E:A5:41:09 add ip udp header dhcpcd_send_packet,270:result:590, ifinde:6 unregister timer:0x2a8bc08 register timer func=0x2699f0 arg=0x2a8d728 name=send_request -> send_request Allocate a new timer Registered timer 0x2a8b910 will expiry in 12 secs timer 0x2a8b910(send_request -> send_request) will expire in 12 secs timer 0x2a8b910(send_request -> send_request) will expire in 11 secs timer 0x2a8b910(send_request -> send_request) will expire in 10 secs timer 0x2a8b910(send_request -> send_request) will expire in 9 secs timer 0x2a8b910(send_request -> send_request) will expire in 8 secs timer 0x2a8b910(send_request -> send_request) will expire in 7 secs timer 0x2a8b910(send_request -> send_request) will expire in 6 secs timer 0x2a8b910(send_request -> send_request) will expire in 5 secs timer 0x2a8b910(send_request -> send_request) will expire in 4 secs timer 0x2a8b910(send_request -> send_request) will expire in 3 secs timer 0x2a8b910(send_request -> send_request) will expire in 2 secs timer 0x2a8b910(send_request -> send_request) will expire in 1 secs timer 0x2a8b910 expired, take action Sending request! state init. make discover make dhcp message, code=1 Insert option(255), len(0) Insert option(53), len(1) Insert max message len (1458) Insert option(57), len(2) Insert client ID Insert option(61), len(7) Insert requested address Insert option(50), len(4) Insert requested options Insert option(55), len(9) Insert hostname Insert option(12), len(16) Insert class ID option Insert option(60), len(13) get_dhcp_msg_len, 303 too small, extend to 548 Sending discover! Send a packet out. add hw header set dst hw addr as: FF:FF:FF:FF:FF:FF src hw addr: 08:5B:0E:A5:41:09 add ip udp header dhcpcd_send_packet,270:result:590, ifinde:6 unregister timer:0x2a8b910 register timer func=0x2698a8 arg=0x2a8d728 name=send_discover -> send_discover Allocate a new timer Registered timer 0x2a8bc08 will expiry in 3 secs timer 0x2a8bc08(send_discover -> send_discover) will expire in 3 secs fd 13 can be read now ###############3Receive packet: len=375 del hw header ether_type:0800 hw addr from: 00:01:5C:86:C8:46 del ip udp header final dhcp message len:333 DHCP Message received. parse dhcp options parse dhcp option buffer (93 bytes) option[53], len:1 option[54], len:4 option[51], len:4 option[1], len:4 option[3], len:4 option[6], len:8 option[15], len:12 option[28], len:4 option[12], len:33 DHCPOFFER received on wan client Xid:0x0ADAABB2 in state:2, pkt Xid:0x0ADAABB2 handle received dhcp options! lease ip:948330C6 lease time: 2689, renew: 0, rebind: 0 unregister timer:0x2a8bc08 Unregistered timer func=0x2698a8 arg=0x2a8d728 name=send_discover -> send_discov er register timer func=0x269598 arg=0x2a8d728 name=dhcp_offer -> state_selecting Allocate a new timer Registered timer 0x2a8bc08 will expiry in 0 secs timer 0x2a8bc08 expired, take action state selecting. make request make dhcp message, code=3 Insert option(255), len(0) Insert option(53), len(1) Insert max message len (1458) Insert option(57), len(2) Insert client ID Insert option(61), len(7) Insert requested address (948330C6) Insert option(50), len(4) Insert server id Insert option(54), len(4) Insert requested options Insert option(55), len(9) Insert hostname Insert option(12), len(16) Insert class ID option Insert option(60), len(13) get_dhcp_msg_len, 309 too small, extend to 548 Sending request! Send a packet out. add hw header set dst hw addr as: FF:FF:FF:FF:FF:FF src hw addr: 08:5B:0E:A5:41:09 add ip udp header dhcpcd_send_packet,270:result:590, ifinde:6 unregister timer:0x2a8bc08 register timer func=0x2699f0 arg=0x2a8d728 name=send_request -> send_request Allocate a new timer Registered timer 0x2a8b910 will expiry in 3 secs timer 0x2a8b910(send_request -> send_request) will expire in 3 secs timer 0x2a8b910(send_request -> send_request) will expire in 2 secs timer 0x2a8b910(send_request -> send_request) will expire in 1 secs timer 0x2a8b910 expired, take action Sending request! Send a packet out. add hw header set dst hw addr as: FF:FF:FF:FF:FF:FF src hw addr: 08:5B:0E:A5:41:09 add ip udp header dhcpcd_send_packet,270:result:590, ifinde:6 unregister timer:0x2a8b910 register timer func=0x2699f0 arg=0x2a8d728 name=send_request -> send_request Allocate a new timer Registered timer 0x2a8bc08 will expiry in 5 secs timer 0x2a8bc08(send_request -> send_request) will expire in 5 secs timer 0x2a8bc08(send_request -> send_request) will expire in 4 secs timer 0x2a8bc08(send_request -> send_request) will expire in 3 secs fd 13 can be read now ###############3Receive packet: len=60 del hw header ether_type:0806 hw addr from: 00:01:5C:86:C8:46 arp packet received, len:46 A ARP packet is received. timer 0x2a8bc08(send_request -> send_request) will expire in 2 secs timer 0x2a8bc08(send_request -> send_request) will expire in 1 secs timer 0x2a8bc08 expired, take action Sending request! Send a packet out. add hw header set dst hw addr as: FF:FF:FF:FF:FF:FF src hw addr: 08:5B:0E:A5:41:09 add ip udp header dhcpcd_send_packet,270:result:590, ifinde:6 unregister timer:0x2a8bc08 register timer func=0x2699f0 arg=0x2a8d728 name=send_request -> send_request Allocate a new timer Registered timer 0x2a8b910 will expiry in 9 secs timer 0x2a8b910(send_request -> send_request) will expire in 9 secs timer 0x2a8b910(send_request -> send_request) will expire in 8 secs timer 0x2a8b910(send_request -> send_request) will expire in 7 secs timer 0x2a8b910(send_request -> send_request) will expire in 6 secs timer 0x2a8b910(send_request -> send_request) will expire in 5 secs timer 0x2a8b910(send_request -> send_request) will expire in 4 secs

Here is what fortinet support and I were able to figure out:  30D discovers DHCP, offer is made, 30D requests and no ACK is sent... most of the time.  Sometimes an ack is sent but from a different IP. Here is a convesion of the logs, formatting is off, sorry.

DHCP Discovery

No. Time Source Destination Protocol Length Info 1 0.000000 198.48.131.148 255.255.255.255 DHCP 590 DHCP Request - Transaction ID 0xe54e6019

Frame 1: 590 bytes on wire (4720 bits), 590 bytes captured (4720 bits) Ethernet II, Src: Fortinet_a5:41:09 (08:5b:0e:a5:41:09), Dst: Broadcast (ff:ff:ff:ff:ff:ff) Internet Protocol Version 4, Src: 198.48.131.148, Dst: 255.255.255.255 User Datagram Protocol, Src Port: 68 (68), Dst Port: 67 (67) Bootstrap Protocol (Request)

No. Time Source Destination Protocol Length Info 2 7.213293 24.65.32.1 255.255.255.255 DHCP 342 DHCP ACK - Transaction ID 0x2c6e0ad2

Frame 2: 342 bytes on wire (2736 bits), 342 bytes captured (2736 bits) Ethernet II, Src: Cadant_86:c8:46 (00:01:5c:86:c8:46), Dst: Broadcast (ff:ff:ff:ff:ff:ff) Internet Protocol Version 4, Src: 24.65.32.1, Dst: 255.255.255.255 User Datagram Protocol, Src Port: 67 (67), Dst Port: 68 (68) Bootstrap Protocol (ACK)

No. Time Source Destination Protocol Length Info 3 7.222009 0.0.0.0 255.255.255.255 DHCP 590 DHCP Discover - Transaction ID 0xe7d8a924

Frame 3: 590 bytes on wire (4720 bits), 590 bytes captured (4720 bits) Ethernet II, Src: Fortinet_a5:41:09 (08:5b:0e:a5:41:09), Dst: Broadcast (ff:ff:ff:ff:ff:ff) Internet Protocol Version 4, Src: 0.0.0.0, Dst: 255.255.255.255 User Datagram Protocol, Src Port: 68 (68), Dst Port: 67 (67) Bootstrap Protocol (Discover)

No. Time Source Destination Protocol Length Info 4 7.234151 198.48.131.129 255.255.255.255 DHCP 375 DHCP Offer - Transaction ID 0xe7d8a924

Frame 4: 375 bytes on wire (3000 bits), 375 bytes captured (3000 bits) Ethernet II, Src: Cadant_86:c8:46 (00:01:5c:86:c8:46), Dst: Broadcast (ff:ff:ff:ff:ff:ff) Internet Protocol Version 4, Src: 198.48.131.129, Dst: 255.255.255.255 User Datagram Protocol, Src Port: 67 (67), Dst Port: 68 (68) Bootstrap Protocol (Offer)

No. Time Source Destination Protocol Length Info 5 7.244463 198.48.131.148 255.255.255.255 DHCP 590 DHCP Request - Transaction ID 0xe7d8a924

Frame 5: 590 bytes on wire (4720 bits), 590 bytes captured (4720 bits) Ethernet II, Src: Fortinet_a5:41:09 (08:5b:0e:a5:41:09), Dst: Broadcast (ff:ff:ff:ff:ff:ff) Internet Protocol Version 4, Src: 198.48.131.148, Dst: 255.255.255.255 User Datagram Protocol, Src Port: 68 (68), Dst Port: 67 (67) Bootstrap Protocol (Request)

No. Time Source Destination Protocol Length Info 6 10.240104 198.48.131.148 255.255.255.255 DHCP 590 DHCP Request - Transaction ID 0xe7d8a924

Frame 6: 590 bytes on wire (4720 bits), 590 bytes captured (4720 bits) Ethernet II, Src: Fortinet_a5:41:09 (08:5b:0e:a5:41:09), Dst: Broadcast (ff:ff:ff:ff:ff:ff) Internet Protocol Version 4, Src: 198.48.131.148, Dst: 255.255.255.255 User Datagram Protocol, Src Port: 68 (68), Dst Port: 67 (67) Bootstrap Protocol (Request)

No. Time Source Destination Protocol Length Info 7 13.240062 198.48.131.148 255.255.255.255 DHCP 590 DHCP Request - Transaction ID 0xe7d8a924

Frame 7: 590 bytes on wire (4720 bits), 590 bytes captured (4720 bits) Ethernet II, Src: Fortinet_a5:41:09 (08:5b:0e:a5:41:09), Dst: Broadcast (ff:ff:ff:ff:ff:ff) Internet Protocol Version 4, Src: 198.48.131.148, Dst: 255.255.255.255 User Datagram Protocol, Src Port: 68 (68), Dst Port: 67 (67) Bootstrap Protocol (Request)

No. Time Source Destination Protocol Length Info 8 19.244059 0.0.0.0 255.255.255.255 DHCP 590 DHCP Discover - Transaction ID 0x7bb43b0b

Frame 8: 590 bytes on wire (4720 bits), 590 bytes captured (4720 bits) Ethernet II, Src: Fortinet_a5:41:09 (08:5b:0e:a5:41:09), Dst: Broadcast (ff:ff:ff:ff:ff:ff) Internet Protocol Version 4, Src: 0.0.0.0, Dst: 255.255.255.255 User Datagram Protocol, Src Port: 68 (68), Dst Port: 67 (67) Bootstrap Protocol (Discover)

No. Time Source Destination Protocol Length Info 9 19.256382 198.48.131.129 255.255.255.255 DHCP 375 DHCP Offer - Transaction ID 0x7bb43b0b

Frame 9: 375 bytes on wire (3000 bits), 375 bytes captured (3000 bits) Ethernet II, Src: Cadant_86:c8:46 (00:01:5c:86:c8:46), Dst: Broadcast (ff:ff:ff:ff:ff:ff) Internet Protocol Version 4, Src: 198.48.131.129, Dst: 255.255.255.255 User Datagram Protocol, Src Port: 67 (67), Dst Port: 68 (68) Bootstrap Protocol (Offer)

No. Time Source Destination Protocol Length Info 10 19.266303 198.48.131.148 255.255.255.255 DHCP 590 DHCP Request - Transaction ID 0x7bb43b0b

Frame 10: 590 bytes on wire (4720 bits), 590 bytes captured (4720 bits) Ethernet II, Src: Fortinet_a5:41:09 (08:5b:0e:a5:41:09), Dst: Broadcast (ff:ff:ff:ff:ff:ff) Internet Protocol Version 4, Src: 198.48.131.148, Dst: 255.255.255.255 User Datagram Protocol, Src Port: 68 (68), Dst Port: 67 (67) Bootstrap Protocol (Request)

[Jeff_S]

Oluf,

I had the engineer I was working with review your post as well as some more logs from my end and he is reporting he has found a known bug that is being working on but didn't go into great detail.  I have requested more information on the bug he is referring to.

[Oluf]

Jeff0123 wrote:

Oluf,

I had the engineer I was working with review your post as well as some more logs from my end and he is reporting he has found a known bug that is being working on but didn't go into great detail.  I have requested more information on the bug he is referring to.

Ok. Let me know if you get any more info on the bug.

[Jeff_the_Network_Guy]

Do you know if this was ever resolved?  I think I'm running into this with a remote sales office.

[Oluf]

Jeff the Network Guy wrote:

Do you know if this was ever resolved?  I think I'm running into this with a remote sales office.

I have not seen the problem after upgrading to v5.4.1