Technical Tip: Understanding Multicast Forwarding Delays in Firewalls Due to kernel and User-Space Coordination

[vpalli]

Description	This article explains the underlying causes of multicast forwarding delays observed on FortiGates. It focuses on the coordination between kernel space and user space processes, especially in Linux/Unix-based systems, such as FortiGate, where multicast protocols, like PIM, are managed by user-space daemons.
Scope	FortiGate.
Solution	On Linux/Unix systems, including FortiGate, multicast routing is generally handled in user space by a routing daemon such as pimd. The kernel manages IGMP and packet forwarding, but the user-space daemon processes PIM signaling. When a multicast packet arrives that needs forwarding, the kernel notifies the PIM daemon, which then sets up the multicast routing state in the kernel through system calls. This process, especially the first time a stream is requested, can take some time, sometimes up to 10 seconds, depending on system load, daemon responsiveness, and PIM behavior. Topology: This 10-second delay is often a combination of Time for IGMP Join to reach the Firewall FGT-B acting as Receiver DR and RP. Time for PIM Join to be generated and reach the RP. Time for PIM Register to get from source to RP. Time required for the user-space daemon (pimd) to install mroute entries in the kernel. This includes informing the kernel whether the FortiGate is acting as the DR/RP and specifying the interface(s) through which traffic should be forwarded. To view the mroute entry as seen and maintained by the user-space daemon pimd, run the following command  get router info multicast pim sparse-mode table IP Multicast Routing Table (*,*,RP) Entries: 0 (*,G) Entries: 21 (S,G) Entries: 7 (S,G,rpt) Entries: 7 FCR Entries: 2 (*, 239.0.1.8) - 3 RP: 10.253.200.1 RPF nbr: 0.0.0.0 RPF idx: None RPF RP: 10.253.200.1, 0, 3, 5, 846931300 Upstream State: JOINED Downstream Expired: 0  Local:      vlan2600      Total: 1  Joined:      Total: 0  Lost assert:      Total: 0 FCR: (10.200.200.7, 239.0.1.8) - 3 RPF nbr: 10.254.200.241 RPF idx: vlan1100 RPF RP: 10.253.200.1, 0, 3, 5, 846931300 RPF Source: 1, 4, 846931300 SPT bit: 0 Upstream State: JOINED Downstream Expired: 0  Local:      Total: 0  Joined:      Total: 0  Lost assert:      Total: 0  Outgoing:      vlan2600      Total: 1 (10.200.200.7, 239.0.1.8, rpt) RP: 10.253.200.1 RPF nbr: 0.0.0.0 RPF idx: None RPF RP: 10.253.200.1, 0, 3, 5, 846931300 RPF Source: 1, 4, 846931300 Upstream State: NOT PRUNED Downstream Expired: 0  Local:      Total: 0  Pruned:      Total: 0  Lost assert:      Total: 0  Outgoing:      vlan2600      Total: 1 For the stream to be forwarded by the FortiGate to the receiver, the kernel must have an mroute entry set up by the user space process 'pimd' with the status set to resolved.  A kernel record is typically created on the RP after the receiver’s IGMP Join reaches the Receiver DR (which is also acting as the RP), the PIM Join is sent from the Receiver DR to the RP, and the stream from the source is received for forwarding to the receiver. To view this kernel record, run the following command Receiver DR# diagnose ip multicast mroute grp=239.0.1.4 src=10.200.200.7 intf=505 flags=(0x10000000)[  ] status=resolved         last_assert=64281 bytes=150840 pkt=2514 wrong_if=0 num_ifs=1         index(ttl)=[511(1),] -----> Index 511 is vlan2600 on Receiver DR. Refer to the output diagnose ip address list The command output shows the multicast source IP, the multicast group, and the interface connected to the receiver. The index value in this output corresponds to the index shown in the 'diagnose ip address list' command, which helps map the index value to the relevant interface name. Multicast forwarding can be delayed if a kernel mroute record is not created or if its status remains ‘unresolved’ on the RP. Once the status changes to ‘resolved’, the RP will begin forwarding the stream to the receiver. The sample output below shows an ‘unresolved’ kernel mroute state, which means the kernel is still waiting for the user-space process pimd to provide details about the multicast DR, the multicast stream, and the receiver interface that should be used for forwarding the stream. Receiver DR# diagnose ip multicast mroute grp=239.0.1.4 src=10.200.200.7 intf=-1 flags=(0x00)[  ] status=unresolved         expires=2021062 qlen=1 A delay in multicast forwarding can be expected each time the kernel mroute record is removed and needs to be re-learned, as per design. This usually happens when the receiver sends an IGMP Leave to the Receiver DR and then later attempts to rejoin the same or a new stream.