Weird issue with 2xPPPoe with SD-WAN

Hi all,

I have a weird problem with 2 PPPoE connection combined under SD-WAN, one interface periodically stops responding from outside even though it is up from the firewall point of view and the route is still present in the table.

My two interfaces are configured as follows:

#10.10.10.1 <- this will be a public IP

config system interface edit "wan1" set vdom "root" set mode pppoe set allowaccess ping set type physical set estimated-upstream-bandwidth 10000 set estimated-downstream-bandwidth 30000 set role wan set snmp-index 1 set username "" set password ENC set dns-server-override disable next end

#10.10.10.2 <- this will be a public IP

config system interface edit "wan2" set vdom "root" set mode pppoe set allowaccess ping https ssh http fgfm set type physical set description "" set estimated-upstream-bandwidth 10000 set estimated-downstream-bandwidth 30000 set role wan set snmp-index 2 set username "" set password ENC next end

they both get a default gw (say 10.10.10.254 - identical for both wan1 & wan2 - same ISP) from the pppoe server and when I do this:

exec ping-options interface wan1

exec ping 10.10.10.254

no response from 10.10.10.254

exec ping-options interface wan2

exec ping 10.10.10.254

get response from 10.10.10.254

5 mins later I can get responses via both wan1 & wan2 and then another 10 mins later it can go weird again, wan2 works all the time wan1 goes dark periodically. IP on wan2 10.10.10.2 I can ping from outside all the time. IP on wan1 10.10.10.1 will be periodically available and when it is I can ping it from my home but quite often it is still not available via 4G from my phone which makes no sense to me.

Routing table will always look like this:

get router info routing-table  all

Routing table for VRF=0

Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP

       O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2

       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area

       * - candidate default

S*      0.0.0.0/0 [1/0] via 10.10.10.254, ppp1

                  [1/0] via 10.10.10.254, ppp2

C       10.10.10.1/32 is directly connected, ppp1

C       10.10.10.2/32 is directly connected, ppp2

C       10.10.10.254/32 is directly connected, ppp2

                        is directly connected, ppp1

C       192.168.0.0/24 is directly connected, lan1

config system virtual-wan-link

    set status enable

    config members

        edit 4

            set interface "wan1"

        next

        edit 2

            set interface "wan2"

        next

    end

end

My ISP claims everything is okay at their end.

I did packet sniffer and when wan1 works I can see packets going into wan1 and then out to wan1, when it doesn't work I can't see any packets hitting wan1. When I do traceroute from outside it successfully hits the gateway (10.10.10.254) no matter but of course doesn't hit wan1 when wan1 "doesn't work". Again, PPPoE doesn't go down, it is always up.

Is there a way to prove that the firewall is not a problem in this equation? Or if there is a known issue how can it be solved?

Thank you.