OT segregation with intervlan traffic

[jfrye]

I am trying to use a fortigate 60F to segregate my OT network, so SCADA from ICS.  I already have switches in place, so trying to use what is there.  Currently testing on the bench, with 2 managed switches connected to the FGT, each switch represents one VLAN, as below:

VLAN 10 - 10.1.0.0/24

VLAN 11 - 10.1.1.0/24

I have assigned physical interfaces (left at 0.0.0.0/0) to VLAN switches (as above). 

Alternatively assigned VLAN interfaces to the physical interfaces (0.0.0.0/0 again).

Both with bidirectional firewall policies.

Using VLAN switches, which was the best result, I can link to one switch, and get to the FGT but not to a device or other switch in the other VLAN; the FGT can ping devices on either VLAN.  What am I missing?  Thanks.

[AEK]

When a devices replies to a ping on the same subnet but doesn't respond to a device on another subnet this usually means the default gateway may not be properly configured on the device.

Can you double check this.

Otherwise you may need packet sniffer on the devices to try understand what is it doing with the lost packets.

[jfrye]

This is setup the way I have other switches setup within my network.

FGT settingsSCADA switch gatewayICS switch gateway

[AEK]

Please share the following cli output:

show system interface 

[jfrye]

FortiGate-60F # show system interface
config system interface
    edit "wan1"
        set vdom "root"
        set mode dhcp
        set allowaccess ping fgfm
        set type physical
        set role wan
        set snmp-index 1
    next
    edit "wan2"
        set vdom "root"
        set mode dhcp
        set allowaccess ping fgfm
        set type physical
        set role wan
        set snmp-index 2
        set macaddr 74:78:a6:dc:83:8f
    next
    edit "dmz"
        set vdom "root"
        set ip 10.10.10.1 255.255.255.0
        set allowaccess ping https fgfm fabric
        set type physical
        set role dmz
        set snmp-index 3
    next
    edit "internal1"
        set vdom "root"
        set type physical
        set snmp-index 4
    next
    edit "internal2"
        set vdom "root"
        set type physical
        set snmp-index 5
    next
    edit "internal3"
        set vdom "root"
        set type physical
        set snmp-index 6
    next
    edit "internal4"
        set vdom "root"
        set type physical
        set snmp-index 7
    next
    edit "internal5"
        set vdom "root"
        set type physical
        set snmp-index 8
    next 
    edit "a"
        set vdom "root"
        set type physical
        set snmp-index 9
    next 
    edit "b"
        set vdom "root"
        set type physical
        set snmp-index 10
    next 
    edit "modem"
        set vdom "root"
        set mode pppoe
        set status down
        set type physical
        set snmp-index 11
    next 
    edit "l2t.root"
        set vdom "root"
        set type tunnel
        set snmp-index 13
    next 
    edit "ssl.root"
        set vdom "root"
        set type tunnel
        set alias "SSL VPN interface"
        set snmp-index 14
    next 
    edit "naf.root"
        set vdom "root"
        set type tunnel
        set src-check disable
        set snmp-index 12
    next 
    edit "internal"
        set vdom "root"
        set ip 192.168.1.99 255.255.255.0
        set allowaccess ping https ssh fgfm fabric
        set type hard-switch
        set alias "Management"
        set stp enable
        set role lan
        set snmp-index 15
    next 
    edit "fortilink"
        set vdom "root"
        set fortilink enable
        set ip 10.255.1.1 255.255.255.0
        set allowaccess ping fabric
        set type aggregate
        set member "a" "b"
        set lldp-reception enable
        set lldp-transmission enable
        set snmp-index 16
    next 
    edit "scda"
        set vdom "root"
        set ip 10.1.0.145 255.255.255.0
        set allowaccess ping https http fgfm fabric
        set type hard-switch
        set device-identification enable
        set lldp-transmission enable
        set role lan
        set snmp-index 17
    next 
    edit "ics"
        set vdom "root"
        set ip 10.1.1.145 255.255.255.0
        set allowaccess ping https http fgfm fabric
        set type hard-switch
        set device-identification enable
        set lldp-transmission enable
        set role lan
        set snmp-index 18
    next 
end      

[AEK]

Please run this debug command while doing the ping test.

diag sniffer packet any "host x.x.x.x and icmp" 4

Also use a packet sniffer on target server to see what it is doing with the echo requests.

[jfrye]

Pinging switch to switch:

FortiGate-60F # diag sniffer packet any 'host 10.1.1.35 and icmp' 4
interfaces=[any]
filters=[host 10.1.1.35 and icmp]
20.976165 scda in 10.1.0.36 -> 10.1.1.35: icmp: echo request
20.977420 ics out 10.1.0.36 -> 10.1.1.35: icmp: echo request
20.977778 ics in 10.1.1.35 -> 10.1.0.36: icmp: echo reply
20.978116 scda out 10.1.1.35 -> 10.1.0.36: icmp: echo reply
21.970243 scda in 10.1.0.36 -> 10.1.1.35: icmp: echo request
21.970288 ics out 10.1.0.36 -> 10.1.1.35: icmp: echo request
21.970713 ics in 10.1.1.35 -> 10.1.0.36: icmp: echo reply
21.970735 scda out 10.1.1.35 -> 10.1.0.36: icmp: echo reply

 Pinging a device on ICS side

FortiGate-60F # diag sniffer packet any 'host 10.1.1.101'
interfaces=[any]
filters=[host 10.1.1.101]
0.445269 arp who-has 10.1.1.65 tell 10.1.1.101
1.054860 10.1.0.36 -> 10.1.1.101: icmp: echo request
1.054993 10.1.0.36 -> 10.1.1.101: icmp: echo request
1.055244 10.1.1.101 -> 10.1.0.36: icmp: echo reply
1.055616 10.1.1.101 -> 10.1.0.36: icmp: echo reply
2.047644 10.1.0.36 -> 10.1.1.101: icmp: echo request
2.047727 10.1.0.36 -> 10.1.1.101: icmp: echo request
2.048139 10.1.1.101 -> 10.1.0.36: icmp: echo reply
2.048170 10.1.1.101 -> 10.1.0.36: icmp: echo reply

[AEK]

You see the echo requests and echo replies are all properly forwarded by the FG. I think there is no issue with the FortiGate.

If you don't receive echo reply on the initiating host then you need packet sniffer on the "initiating" client to see what is it doing with the received echo replies.

[jfrye]

This is packet sniffer on the initiating host

FortiGate-60F # diag sniffer packet any 'host 10.1.0.36 and icmp' 4
interfaces=[any]
filters=[host 10.1.0.36 and icmp]
27.081569 scda in 10.1.0.36 -> 10.1.1.35: icmp: echo request
27.082727 ics out 10.1.0.36 -> 10.1.1.35: icmp: echo request
27.083085 ics in 10.1.1.35 -> 10.1.0.36: icmp: echo reply
27.083114 scda out 10.1.1.35 -> 10.1.0.36: icmp: echo reply
28.075276 scda in 10.1.0.36 -> 10.1.1.35: icmp: echo request
28.075341 ics out 10.1.0.36 -> 10.1.1.35: icmp: echo request
28.075757 ics in 10.1.1.35 -> 10.1.0.36: icmp: echo reply
28.075783 scda out 10.1.1.35 -> 10.1.0.36: icmp: echo reply