Issue multiple wan - multiple IP - multiple port forwarding

[looney1977]

Hi,

I'm quite new in Fortinet world, I'm currently facing some difficulties to configure a fortinet correctly.

It's a Fortinet 80E with release 5.6.

Here is my actual setup :

Got 3 WAN interface configured in a virtual-wan-link.

One of those WAN interface has multiple subnet configured because it's also used to accept incoming trafic, it's configured as second interface in the virtual-wan-link interface.

Outgoing trafic is working fine over the 3 interfaces (failover is working properly)

For my incoming traffic I created all VIP necessary, and all policy-rules. But my server aren't responding. I guess it's caused by the default route to the virtual-wan-link. So how can I configure another default route specific for the VIP and this interface when it's used for incoming trafic ?

Thanks for your tips.

Dominique

[ede_pfau]

hi,

and welcome to the forums.

Your problem should only occur on incoming traffic, not outgoing from the serves nor reply traffic. Is that right?

I don't see how the default route would influence incoming traffic - maybe you could give us more details on that idea.

Could you please post the config of one VIP, and the WAN llink config, from CLI (in text form)? So we can see the implementation details.

[looney1977]

Hi Ede,

Thanks a lot for your quick answer.

I actually have a lab running, and I'm trying to work with one vip for test :

Got a WAN emulated network with one device, from this device I'm constantly pinging yy.yyy.yyy.yy1, and I'm also trying to reach port 80 from yy.yyy.yyy.yy1 address.

I've got a tcpdump running on the destination internal server, but no packet are found on port 80

I've got a diag sniffer packet on the correct interface, I can see incoming request on port 80 and icmp packetq, but no reply are emitted.

My suspiscion is that the fortigate answers but to the incorrect default gateway which should be actually the multi-wan-link interface (active wan interface).

Thanks a lot for you time.

here is the dump you're asking for (sry for evident reasons, I replaced public IP by letters) :

config system interface
    edit "ORANGE_SDSL"
        set vdom "root"
        set ip xx.xxx.xxx.xx1 255.255.255.248
        set allowaccess ping
        set role wan
        set snmp-index 29
        set secondary-IP enable
        set interface "port5"
        set vlanid 61
        config secondaryip
            edit 1
                set ip xx.xxx.xxx.xx2 255.255.255.248
            next
            edit 2
                set ip xx.xxx.xxx.xx3 255.255.255.248
            next
            edit 3
                set ip xx.xxx.xxx.xx4 255.255.255.248
            next
            edit 4
                set ip xx.xxx.xxx.xx5 255.255.255.248
            next
            edit 5
                set ip yy.yyy.yyy.yy1 255.255.255.248
                set allowaccess ping
            next
            edit 6
                set ip yy.yyy.yyy.yy2 255.255.255.248
            next
            edit 7
                set ip yy.yyy.yyy.yy3 255.255.255.248
            next
            edit 8
                set ip yy.yyy.yyy.yy4 255.255.255.248
            next
            edit 9
                set ip yy.yyy.yyy.yy5 255.255.255.248
            next
        end
    next
end

config system virtual-wan-link
    set status enable
    config members
        edit 1
            set interface "wan1"
            set gateway 10.75.176.1
        next
        edit 2
            set interface "PROXIMUS_VDSL"
            set gateway 192.168.250.1
        next
        edit 3
            set interface "ORANGE_SDSL"
            set gateway xx.xxx.xxx.xx0
        next
    end
end

config firewall vip
    edit "test-vip-ws002-vm-port-80"
        set uuid bcafc936-5a5f-51e7-acfa-36481a536169
        set extip yy.yyy.yyy.yy1
        set extintf "ORANGE_SDSL"
        set portforward enable
        set mappedip "172.16.2.222"
        set extport 80
        set mappedport 80
    next
end

[ede_pfau]

OK, so you've got a VLAN port, sub-port of "port5" physical port, with 9 secondary addresses...didn't know that a VLAN port could have those.

VIP: remove the port forwarding to enable ping across the VIP. You can narrow down the port with service=HTTP in the policy. As ICMP is portless it will not be forwarded by a port-forwarding VIP (makes sense, right?).

Now, how do you use the sniffer? You should see pings on port "ORANGE_SDSL" and on the DMZ port. That is, requests. Then you should have replies on the DMZ port - so your server has seen the pings and is responding.

To see how the NAT is working use "diag deb flow" with filters, see the forums for detailed commands. It's one of the most used diag commands after 'sniffer'.

I still can't make out where the routing comes in - what default route does the server have? You could enable NAT in the incoming policy to make the traffic appear as 'internal'. If this helps your routing is incorrect, but probably not on the FGT.

[looney1977]

We've placed a switch between provider interfaces and firewall interfaces to increase redundancy and failover mechanism. But that's not the issue, that part is working, the switch is transparent.

Indeed ICMP is portless, also I'm not trying that my ICMP packet is reaching my server, only checking my public interface is knowing where to send this reply of the ICMP packet.

The VIP is there to allow port 80 to be routed to internal server.

diag sniffer packet ORANGE_SDSL gives as answer :

6.535654 12.10.10.2 -> yy.yyy.yyy.yy2: icmp: echo request

Like suggested I went deeper via diag deb flow (thanks for the tip!) and found some event with msg="reverse path check fail, drop"

So like found on this link : http://kb.fortinet.com/kb/documentLink.do?externalID=FD30543 I enabled asymroute.

Now I'm a little bit further, but still doesn't fully works :(  I can now see my packet coming to my internal server, and it's answering but like it seems to be shown in the trace, answers are routed to the wrong interface :(

And ICMP packet aren't working either.

id=20085 trace_id=74 func=print_pkt_detail line=5319 msg="vd-root received a packet(proto=6, 12.10.10.2:59755->yy.yyy.yyy.yy2:80) from ORANGE_SDSL. flag , seq 3668301584, ack 0, win 65535"
id=20085 trace_id=74 func=init_ip_session_common line=5475 msg="allocate a new session-008b9087"
id=20085 trace_id=74 func=fw_pre_route_handler line=182 msg="VIP-172.16.2.222:80, outdev-ORANGE_SDSL"
id=20085 trace_id=74 func=__ip_session_run_tuple line=3140 msg="DNAT yy.yyy.yyy.yy2:80->172.16.2.222:80"
id=20085 trace_id=74 func=vf_ip_route_input_common line=2578 msg="find a route: flag=00000000 gw-172.16.2.222 via DMZ-OPERATIONAL"
id=20085 trace_id=74 func=fw_forward_handler line=710 msg="Allowed by Policy-69:"
id=20085 trace_id=75 func=print_pkt_detail line=5319 msg="vd-root received a packet(proto=6, 172.16.2.222:80->12.10.10.2:59755) from DMZ-OPERATIONAL. flag [S.], seq 4088857877, ack 3668301585, win 29200"
id=20085 trace_id=75 func=resolve_ip_tuple_fast line=5394 msg="Find an existing session, id-008b9087, reply direction"
id=20085 trace_id=75 func=vf_ip_route_input_common line=2578 msg="find a route: flag=04000000 gw-192.168.250.1 via PROXIMUS_VDSL"
id=20085 trace_id=75 func=npu_handle_session44 line=1067 msg="Trying to offloading session from DMZ-OPERATIONAL to PROXIMUS_VDSL, skb.npu_flag=00000400 ses.state=00010204 ses.npu_state=0x00000000"
id=20085 trace_id=75 func=ip_session_install_npu_session line=347 msg="npu session installation succeeded"
id=20085 trace_id=75 func=__ip_session_run_tuple line=3126 msg="SNAT 172.16.2.222->yy.yyy.yyy.yy2:80"

Should I now add extended routing, like policy routing ?

Br,

Dominique

[Uwe_Sommerfeld]

Could you add your routing configuration plus a get router info routing-table all for us?

[looney1977]

Hi,

Here is my routing table :

Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default

S*      0.0.0.0/0 [1/0] via 192.168.250.1, PROXIMUS_VDSL
S       10.75.179.0/24 [10/0] via 172.16.1.1, ROUTING
S       10.75.180.0/23 [10/0] via 172.16.1.1, ROUTING
S       10.75.182.0/23 [10/0] via 172.16.1.1, ROUTING
C       yy.yyy.yyy.yy0/29 is directly connected, ORANGE_SDSL
                          is directly connected, ORANGE_SDSL
                          is directly connected, ORANGE_SDSL
                          is directly connected, ORANGE_SDSL
                          is directly connected, ORANGE_SDSL
C       xx.xxx.xxx.xx0/29 is directly connected, ORANGE_SDSL
                         is directly connected, ORANGE_SDSL
                         is directly connected, ORANGE_SDSL
                         is directly connected, ORANGE_SDSL
                         is directly connected, ORANGE_SDSL
C       172.16.1.0/24 is directly connected, ROUTING
C       172.16.2.0/24 is directly connected, DMZ-OPERATIONAL
C       172.99.1.248/29 is directly connected, PROXIMUS_ADSL
C       192.168.250.0/29 is directly connected, PROXIMUS_VDSL


Br,

Dominique

[Uwe_Sommerfeld]

Hi Dominique,

I would try this - Change your WAN Routing configuration so that all WAN Interfaces have the same administrative distance. Your Primary WAN Proximus should have priority 0 and the other WAN(s) should have a higher Priority number, meaning lower priority.

What does happen is that all Default routes will be included in the Routing table. New Outbound sessions will still be only on Proximus as Long as it exists in the Routing table.

This way reverse Routing could see the other Default routes and your web Servers should be accessible on those IP addresses.

Uwe

[ede_pfau]

Please, do not enable asymroute!

With this enabled you discard stateful inspection, rendering the firewall more or less useless.

If you have an issue with routing, fix it. 'Reverse path check' clearly indicates that there is no return route for this traffic, probably because it's going out one interface and coming in the other.