Cisco IOS to Fortinet - establishes but one way traffic

[tim080772]

Hi all,

I’ve been banging my head and have spent hours looking at this and I just can’t find the issue.  Below is a simple diagram of the setup.

Cisco Router GSM Interface - Carrier – Internet - Fortigate      Cisco Switch

10.90.60.1    B.B.B.B                                A.A.A.A        10.90.200.254

The problem I have is that the tunnel comes up and all looks great but traffic doesn’t go over the tunnel, well that’s also not quite true, it goes one way but doesn’t make it all the way back.  I am not sure if this is a Cisco or Fortigate issue, but I know I’m stumped.

If I ping from the router on the left (10.90.60.1) to the Switch on the right (10.90.200.254) I can see in the debug on the switch the ping coming in and the echo response being sent.  In the Fortigate I can see both the ping and echo packets and according to the firewall rules they are allowed through, I can even see it in the diag debug flow, but the echo replies don’t turn up at the Cisco router and I don’t even see the packets come in.  Any advice greatly appreciated!

Sanitised data:

A.A.A.A = Fortigate outside Interface

B.B.B.B = 3G router DHCP provided address

C.C.C.C=Internet Gateway address

Fortigate debug output

TLFW01 # diagnose debug enable

TLFW01 # diagnose debug disableike 0:3G Routers_0: link is idle 3 A.A.A.A->B.B.B.B:500 dpd=1 seqno=22

ike 0:3G Routers_0:20007: send IKEv1 DPD probe, seqno 34

ike 0:3G Routers_0:20007: enc D739053A31EEDA7FD9ADA680A81FCA7308100501DD20B3B4000000500B00001407A593C22249BFCDB608454035607D47000000200000000101108D28D739053A31EEDA7FD9ADA680A81FCA7300000022

ike 0:3G Routers_0:20007: out D739053A31EEDA7FD9ADA680A81FCA7308100501DD20B3B40000005405806CC3E606A618249217A38584E728B4F992212D64A7AADF6F295946E55D340FAF7B8866F4EE9C715C9C0B5455140BDEA20C8A54E57F61

ike 0:3G Routers_0:20007: sent IKE msg (R-U-THERE): A.A.A.A:500->B.B.B.B:47886, len=84, id=d739053a31eeda7f/d9ada680a81fca73:dd20b3b4

ike 0: comes B.B.B.B:47886->A.A.A.A:500,ifindex=3....

ike 0: IKEv1 exchange=Informational id=d739053a31eeda7f/d9ada680a81fca73:0a52c0fd len=84

ike 0: in D739053A31EEDA7FD9ADA680A81FCA73081005010A52C0FD00000054334A734A02EC1BA04761CBB4F5593E0EB25C13610AE03953B0B5E42D5F1F59919240756D03D7613494DC661F4F86E1BBA5DD03187469BB4F

ike 0:3G Routers_0:20007: dec D739053A31EEDA7FD9ADA680A81FCA73081005010A52C0FD000000540B000014E27287E2B3E4504827C8155AF69FCCF5000000200000000101108D29D739053A31EEDA7FD9ADA680A81FCA730000002200000000

ike 0:3G Routers_0:20007: notify msg received: R-U-THERE-ACK

id=36871 trace_id=75 func=resolve_ip_tuple_fast line=3799 msg="vd-root received a packet(proto=1, 10.90.60.1:109->10.90.10.254:8) from 3G Routers_0."

id=36871 trace_id=75 func=resolve_ip_tuple line=3939 msg="allocate a new session-01bff608"

id=36871 trace_id=75 func=vf_ip4_route_input line=1601 msg="find a route: gw-10.90.200.254 via internal"

id=36871 trace_id=75 func=fw_forward_handler line=555 msg="Allowed by Policy-13:"

id=36871 trace_id=76 func=resolve_ip_tuple_fast line=3799 msg="vd-root received a packet(proto=1, 10.90.10.254:109->10.90.60.1:0) from internal."

id=36871 trace_id=76 func=resolve_ip_tuple_fast line=3839 msg="Find an existing session, id-01bff608, reply direction"

id=36871 trace_id=76 func=vf_ip4_route_input line=1601 msg="find a route: gw-10.90.60.1 via 3G Routers_0"

id=36871 trace_id=76 func=ipsecdev_hard_start_xmit line=121 msg="enter IPsec interface-3G Routers_0"

id=36871 trace_id=76 func=esp_output4 line=519 msg="encrypted, and send to B.B.B.B with source A.A.A.A"

id=36871 trace_id=76 func=ipsec_output_finish line=210 msg="send to C.C.C.C via intf-wan1"

id=36871 trace_id=77 func=resolve_ip_tuple_fast line=3799 msg="vd-root received a packet(proto=1, 10.90.60.1:109->10.90.10.254:8) from 3G Routers_0."

id=36871 trace_id=77 func=resolve_ip_tuple_fast line=3839 msg="Find an existing session, id-01bff608, original direction"

id=36871 trace_id=78 func=resolve_ip_tuple_fast line=3799 msg="vd-root received a packet(proto=1, 10.90.10.254:109->10.90.60.1:0) from internal."

id=36871 trace_id=78 func=resolve_ip_tuple_fast line=3839 msg="Find an existing session, id-01bff608, reply direction"

id=36871 trace_id=78 func=ipsecdev_hard_start_xmit line=121 msg="enter IPsec interface-3G Routers_0"

id=36871 trace_id=78 func=esp_output4 line=519 msg="encrypted, and send to B.B.B.B with source A.A.A.A"

id=36871 trace_id=78 func=ipsec_output_finish line=210 msg="send to C.C.C.C via intf-wan1"

id=36871 trace_id=79 func=resolve_ip_tuple_fast line=3799 msg="vd-root received a packet(proto=1, 10.90.60.1:109->10.90.10.254:8) from 3G Routers_0."

id=36871 trace_id=79 func=resolve_ip_tuple_fast line=3839 msg="Find an existing session, id-01bff608, original direction"

id=36871 trace_id=80 func=resolve_ip_tuple_fast line=3799 msg="vd-root received a packet(proto=1, 10.90.10.254:109->10.90.60.1:0) from internal."

id=36871 trace_id=80 func=resolve_ip_tuple_fast line=3839 msg="Find an existing session, id-01bff608, reply direction"

id=36871 trace_id=80 func=ipsecdev_hard_start_xmit line=121 msg="enter IPsec interface-3G Routers_0"

id=36871 trace_id=80 func=esp_output4 line=519 msg="encrypted, and send to B.B.B.B with source A.A.A.A"

id=36871 trace_id=80 func=ipsec_output_finish line=210 msg="send to C.C.C.C via intf-wan1"

id=36871 trace_id=81 func=resolve_ip_tuple_fast line=3799 msg="vd-root received a packet(proto=1, 10.90.60.1:109->10.90.10.254:8) from 3G Routers_0."

id=36871 trace_id=81 func=resolve_ip_tuple_fast line=3839 msg="Find an existing session, id-01bff608, original direction"

id=36871 trace_id=82 func=resolve_ip_tuple_fast line=3799 msg="vd-root received a packet(proto=1, 10.90.10.254:109->10.90.60.1:0) from internal."

id=36871 trace_id=82 func=resolve_ip_tuple_fast line=3839 msg="Find an existing session, id-01bff608, reply direction"

id=36871 trace_id=82 func=ipsecdev_hard_start_xmit line=121 msg="enter IPsec interface-3G Routers_0"

id=36871 trace_id=82 func=esp_output4 line=519 msg="encrypted, and send to B.B.B.B with source A.A.A.A"

id=36871 trace_id=82 func=ipsec_output_finish line=210 msg="send to C.C.C.C via intf-wan1"

id=36871 trace_id=83 func=resolve_ip_tuple_fast line=3799 msg="vd-root received a packet(proto=1, 10.90.60.1:109->10.90.10.254:8) from 3G Routers_0."

id=36871 trace_id=83 func=resolve_ip_tuple_fast line=3839 msg="Find an existing session, id-01bff608, original direction"

id=36871 trace_id=84 func=resolve_ip_tuple_fast line=3799 msg="vd-root received a packet(proto=1, 10.90.10.254:109->10.90.60.1:0) from internal."

id=36871 trace_id=84 func=resolve_ip_tuple_fast line=3839 msg="Find an existing session, id-01bff608, reply direction"

id=36871 trace_id=84 func=ipsecdev_hard_start_xmit line=121 msg="enter IPsec interface-3G Routers_0"

id=36871 trace_id=84 func=esp_output4 line=519 msg="encrypted, and send to B.B.B.B with source A.A.A.A"

id=36871 trace_id=84 func=ipsec_output_finish line=210 msg="send to C.C.C.C via intf-wan1"

ike 0:3G Routers_0: link is idle 3 A.A.A.A->B.B.B.B:500 dpd=1 seqno=23

ike 0:3G Routers_0:20007: send IKEv1 DPD probe, seqno 35

ike 0:3G Routers_0:20007: enc D739053A31EEDA7FD9ADA680A81FCA7308100501AE753898000000500B000014CAEC6E94CCBE42E8FE84DB26FC682F5F000000200000000101108D28D739053A31EEDA7FD9ADA680A81FCA7300000023

ike 0:3G Routers_0:20007: out D739053A31EEDA7FD9ADA680A81FCA7308100501AE753898000000544C0B895F46D3B398652CCB5FB40F4AF822D182B6CB2592568873AAA2BB5708ABE5A6202680D3BCA6E65C3F187F5722FE66DD8EC70EC50919

ike 0:3G Routers_0:20007: sent IKE msg (R-U-THERE): A.A.A.A:500->B.B.B.B:47886, len=84, id=d739053a31eeda7f/d9ada680a81fca73:ae753898

ike 0: comes B.B.B.B:47886->A.A.A.A:500,ifindex=3....

ike 0: IKEv1 exchange=Informational id=d739053a31eeda7f/d9ada680a81fca73:375d0fe5 len=84

ike 0: in D739053A31EEDA7FD9ADA680A81FCA7308100501375D0FE500000054A5D62B09881A501682194A75A4BBB4F15EB46D91B52EE126FA9D5E0313B3AD0400E047197FDAFC2730D32BE8349ED868BE6C461C24C48C29

ike 0:3G Routers_0:20007: dec D739053A31EEDA7FD9ADA680A81FCA7308100501375D0FE5000000540B0000148C7E44C56879F207594C0E444C496049000000200000000101108D29D739053A31EEDA7FD9ADA680A81FCA730000002300000000

ike 0:3G Routers_0:20007: notify msg received: R-U-THERE-ACK

 

 

TLFW01 #

TLFW01 #

TLFW01 # diagnose debug disable

Fortigate Firewall Rules

TLFW01 # show firewall policy

config firewall policy

    edit 13

        set srcintf "3G Routers"

        set dstintf "internal"

            set srcaddr "3GDEMORT01"

            set dstaddr "All"

        set action accept

        set schedule "always"

            set service "ANY"

        set logtraffic enable

    next

    edit 17

        set srcintf "internal"

        set dstintf "3G Routers"

            set srcaddr "All"

            set dstaddr "3GDEMORT01"

        set action accept

        set schedule "always"

            set service "ANY"

        set logtraffic enable

    next

TLFW01 # show firewall address

config firewall address

    edit "3GDEMORT01"

        set subnet 10.90.60.0 255.255.255.0

    next

end

TLFW01 # show sys interface

config system interface

    edit "wan1"

        set vdom "root"

        set ip A.A.A.A 255.255.255.248

        set allowaccess ping https

        set type physical

    next

    edit "internal"

        set vdom "root"

        set ip 10.90.200.250 255.255.255.0

        set allowaccess ping https ssh snmp telnet fgfm

        set type physical

        set explicit-web-proxy enable

        set alias "TestLabCommsLAN"

    next

    edit "3G Routers"

        set vdom "root"

        set type tunnel

        set interface "wan1"

    next

end

Cisco debug output

3GDEMORT01#ping 10.90.10.254 source vlan 1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.90.10.254, timeout is 2 seconds:

Packet sent with a source address of 10.90.60.1

*Aug  5 22:31:56.459: IP: s=10.90.60.1 (local), d=10.90.10.254, len 100, local feature

*Aug  5 22:31:56.459:     ICMP type=8, code=0, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:31:56.459: FIBipv4-packet-proc: route packet from (local) src 10.90.60.1 dst 10.90.10.254

*Aug  5 22:31:56.459: FIBfwd-proc: packet routed by adj to Cellular0 0.0.0.0

*Aug  5 22:31:56.459: FIBipv4-packet-proc: packet routing succeeded

*Aug  5 22:31:56.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, sending

*Aug  5 22:31:56.459:     ICMP type=8, code=0

*Aug  5 22:31:56.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:31:56.459:     ICMP type=8, code=0, Post-routing NAT Outside(24), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:31:56.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:31:56.459:     ICMP type=8, code=0, Common Flow Table(27), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:31:56.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:31:56.459:     ICMP type=8, code=0, Stateful Inspection(28), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:31:56.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:31:56.459:     ICMP type=8, code=0, IPSec output classification(34), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:31:56.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:31:56.463:     ICMP type=8, code=0, NAT ALG proxy(59), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:31:56.463: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:31:56.46.3:     ICMP type=8, code=0, packet consumed, IPSec: to crypto engine(74), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:31:58.459: IP: s=10.90.60.1 (local), d=10.90.10.254, len 100, local feature

*Aug  5 22:31:58.459:     ICMP type=8, code=0, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:31:58.459: FIBipv4-packet-proc: route packet from (local) src 10.90.60.1 dst 10.90.10.254

*Aug  5 22:31:58.459: FIBfwd-proc: packet routed by adj to Cellular0 0.0.0.0

*Aug  5 22:31:58.459: FIBipv4-packet-proc: packet routing succeeded

*Aug  5 22:31:58.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, sending

*Aug  5 22:31:58.459:     ICMP type=8, code=0

*Aug  5 22:31:58.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:31:58.459:     ICMP type=8, code=0, Post-routing NAT Outside(24), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:31:58.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:31:58.459:     ICMP type=8, code=0, Common Flow Table(27), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:31:58.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:31:58.459:     ICMP type=8, code=0, Stateful Inspection(28), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:31:58.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:31:58.459:     ICMP type=8, code=0, IPSec output classification(34), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:31:58.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len. 100, output feature

*Aug  5 22:31:58.459:     ICMP type=8, code=0, NAT ALG proxy(59), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:31:58.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:31:58.459:     ICMP type=8, code=0, packet consumed, IPSec: to crypto engine(74), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:00.459: IP: s=10.90.60.1 (local), d=10.90.10.254, len 100, local feature

*Aug  5 22:32:00.459:     ICMP type=8, code=0, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:00.459: FIBipv4-packet-proc: route packet from (local) src 10.90.60.1 dst 10.90.10.254

*Aug  5 22:32:00.459: FIBfwd-proc: packet routed by adj to Cellular0 0.0.0.0

*Aug  5 22:32:00.459: FIBipv4-packet-proc: packet routing succeeded

*Aug  5 22:32:00.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, sending

*Aug  5 22:32:00.459:     ICMP type=8, code=0

*Aug  5 22:32:00.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:00.459:     ICMP type=8, code=0, Post-routing NAT Outside(24), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:00.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:00.459:     ICMP type=8, code=0, Common Flow Table(27), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:00.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:00.459:     ICMP type=8, code=0, Stateful Inspection(28), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:00.459: IP: s=10.90.60.1 (local), d=10..90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:00.459:     ICMP type=8, code=0, IPSec output classification(34), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:00.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:00.459:     ICMP type=8, code=0, NAT ALG proxy(59), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:00.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:00.459:     ICMP type=8, code=0, packet consumed, IPSec: to crypto engine(74), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:02.459: IP: s=10.90.60.1 (local), d=10.90.10.254, len 100, local feature

*Aug  5 22:32:02.459:     ICMP type=8, code=0, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:02.459: FIBipv4-packet-proc: route packet from (local) src 10.90.60.1 dst 10.90.10.254

*Aug  5 22:32:02.459: FIBfwd-proc: packet routed by adj to Cellular0 0.0.0.0

*Aug  5 22:32:02.459: FIBipv4-packet-proc: packet routing succeeded

*Aug  5 22:32:02.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, sending

*Aug  5 22:32:02.459:     ICMP type=8, code=0

*Aug  5 22:32:02.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:02.459:     ICMP type=8, code=0, Post-routing NAT Outside(24), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:02.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:02.459:     ICMP type=8, code=0, Common Flow Table(27), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:02..459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:02.459:     ICMP type=8, code=0, Stateful Inspection(28), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:02.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:02.459:     ICMP type=8, code=0, IPSec output classification(34), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:02.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:02.459:     ICMP type=8, code=0, NAT ALG proxy(59), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:02.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:02.459:     ICMP type=8, code=0, packet consumed, IPSec: to crypto engine(74), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:04.459: IP: s=10.90.60.1 (local), d=10.90.10.254, len 100, local feature

*Aug  5 22:32:04.459:     ICMP type=8, code=0, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:04.459: FIBipv4-packet-proc: route packet from (local) src 10.90.60.1 dst 10.90.10.254

*Aug  5 22:32:04.459: FIBfwd-proc: packet routed by adj to Cellular0 0.0.0.0

*Aug  5 22:32:04.459: FIBipv4-packet-proc: packet routing succeeded

*Aug  5 22:32:04.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, sending

*Aug  5 22:32:04.459:     ICMP type=8, code=0

*Aug  5 22:32:04.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:04.459:     ICMP type=8, code=0, Post-routing NAT Outside(24), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FAL.

Success rate is 0 percent (0/5)

3GDEMORT01#SE

*Aug  5 22:32:04.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:04.459:     ICMP type=8, code=0, Common Flow Table(27), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:04.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:04.459:     ICMP type=8, code=0, Stateful Inspection(28), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:04.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:04.459:     ICMP type=8, code=0, IPSec output classification(34), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:04.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:04.459:     ICMP type=8, code=0, NAT ALG proxy(59), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Aug  5 22:32:04.459: IP: s=10.90.60.1 (local), d=10.90.10.254 (Cellular0), len 100, output feature

*Aug  5 22:32:04.459:     ICMP type=8, code=0, packet consumed, IPSec: to crypto engine(74), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSEno deb all

All possible debugging has been turned off

3GDEMORT01#

[emnoc]

The diag debug flow command is your best friend. I would start  at that 1st and see what is happening and if your matching the suspected policy-id.

[tim080772]

I thought that's what I had included.... As I read this output, the packet was seen, encrypted and passed out to the Internet gateway, have I read it right ?

id=36871 trace_id=76 func=resolve_ip_tuple_fast line=3799 msg="vd-root received a packet(proto=1, 10.90.10.254:109->10.90.60.1:0) from internal."
id=36871 trace_id=76 func=resolve_ip_tuple_fast line=3839 msg="Find an existing session, id-01bff608, reply direction"
id=36871 trace_id=76 func=vf_ip4_route_input line=1601 msg="find a route: gw-10.90.60.1 via 3G Routers_0"
id=36871 trace_id=76 func=ipsecdev_hard_start_xmit line=121 msg="enter IPsec interface-3G Routers_0"
id=36871 trace_id=76 func=esp_output4 line=519 msg="encrypted, and send to B.B.B.B with source A.A.A.A"
id=36871 trace_id=76 func=ipsec_output_finish line=210 msg="send to C.C.C.C via intf-wan1"

[vjoshi_FTNT]

Hello, Yes, the debug flow command output shows that the request hit the correct policy and also the return traffic was encrypted and sent out. In this case, I would recommend doing a sniff as below : # diag sniffer packet any 'host B.B.B.B and esp' 4 Then try to initiate another ping from .60.1 towards Fortigate You may have to play with the size of the ICMP. Idea is to see, if the Fortigate sends the ESP packet out of its WAN interface(you may see several ESP packets, but you can differentiate the ICMP with the size at the end).

If Fortigate is sending the ESP packet out, then, you need to check with the upstream.

[anil_nayak_FTNT]

Hello,

Already quite a few good suggestions already suggested i can see, especially by using a size of the ICMP. 

1. initiate ping from cisco client with size set for example as 800

 ping -l 800 <FGT-server ip address> check on FGT firewall any packet received with size between 800-900(ping size + ESP header + IP pheader)

ICMP-reply size will remain the same (800-900), hence on the cisco end ESP capture you can check if you find any ESP packet with size 800-900 byyes

2. Already the debug flow command output shows that the request hit the correct policy and also the return traffic was encrypted and sent out.

On FGT when pinging from cisco to fgt host, run below command

# diag sniffer packet any 'host B.B.B.B and esp' 4 0 a  You can convert it in wire-shark format and check if there are any missing esp.sequence on the FGT, if none. That would mean that FGT is sending the ESP out. There after you can take a similar ESP capture on cisco and check if any missing ESP. 

If yes--> issue probably with ISP

If no--> decrypt the ESP using the key set and  check if any missing icmp.reply using icmp.seq 

Thanks

Anil