VPN tunnel does not come up: unexpected payload type 11

[Sidlahar]

Hi all,

we are migrating our VPN tunnel from an Juniper ISG 1000 Cluster to a Fortigate 500D Cluster. I have trouble with one Tunnel to an AVM Fritzbox 3490. The tunnel between ISG and Fritzbox works fine. At the Fritzbox we only Change the tunnel peer IP-Address, all other parameters  has been left unchanged. On the Fortigate 500D we set up the tunnel with the same parameters like the ISG, but the Phase1 does not come up. I can see the message "unexpected payload type 11" but have found nothing what this error might be triggered from.

ike 0:P1_ProSicherh-P:1304428: initiator: main mode get 1st response... ike 0:P1_ProSicherh-P:1304428: unexpected payload type 11

As the Fritzbox is not been under our Administration, it is not so easy to do any changes at this side.

Have any one an idea, what is the origin of this message?

Thanks and best regards!

Sidlahar

[MrSinners]

Not sure about the exact origin but it looks like an IKE mismatch in the P1 negotiation so it could be multiple things.

Is it matching the correct Phase 1? What if you try aggressive mode? Is there more debugging output? Is the key life time correct on both sides? And while I doubt it's Phase 2 settings related, did you add the quick mode selectors? And perhaps try to turn of PFS if it's enabled.

[rwpatterson]

If you can get the Juniper config and post (with the peer and local subnets removed, of course) and do the same with the FGT config, perhaps we can give you better guidance.

[Sidlahar]

Hi,

here are the configs of ISG and 500D, I hope there is nothing missing.

Thanks

Sidlahar

ISG:

set ike gateway "ProSicherheit" address X.X.X.X Main outgoing-interface "ethernet1/2.517" preshare "tz6fpqjwN+CfaesHyTCoq2C1/En/QNKS8+ztzItJImVnZFjyzXrUCvg=" proposal "pre-g2-aes256-sha1" set vpn "ProSicherheit" gateway "ProSicherheit" no-replay tunnel idletime 0 proposal "g2-esp-aes256-sha1" set vpn "ProSicherheit" id 0x20346 bind interface tunnel.146 set vpn "ProSicherheit" proxy-id local-ip X.X.X.X/16 remote-ip X.X.X.X/24 "ANY" set ike p1-proposal "pre-g2-aes256-sha1-lt1h" preshare group2 esp aes256 sha-1 second 3600 set ike p2-proposal "g2-esp-aes256-sha1" group2 esp aes256 sha-1 second 3600

500D:

    edit "P1_ProSicherh-P"         set vdom "root"         set dhcp-relay-service disable         set ip 0.0.0.0 0.0.0.0         unset allowaccess         set arpforward enable         set broadcast-forward disable         set bfd global         set l2forward disable         set icmp-redirect enable         set vlanforward disable         set stpforward disable         set ips-sniffer-mode disable         set ident-accept disable         set ipmac disable         set status up         set netbios-forward disable         set wins-ip 0.0.0.0         set type tunnel         set netflow-sampler disable         set sflow-sampler disable         set sample-rate 2000         set polling-interval 20         set sample-direction both         set explicit-web-proxy disable         set explicit-ftp-proxy disable         set tcp-mss 0         set inbandwidth 0         set outbandwidth 0         set spillover-threshold 0         set weight 0         set external disable         set remote-ip 0.0.0.0         set description ''         set alias ''         set security-mode none         set listen-forticlient-connection disable         set snmp-index 69             config ipv6                 set ip6-mode static                 unset ip6-allowaccess                 set ip6-reachable-time 0                 set ip6-retrans-time 0                 set ip6-hop-limit 0                 set ip6-address ::/0                 set ip6-send-adv disable                 set autoconf disable                 set dhcp6-relay-service disable             end         unset dhcp-relay-ip         set dhcp-relay-type regular         set interface "untrust-rku-fw"     next   edit "P1_ProSicherh-P"     set type static         set interface "untrust-rku-fw"         set ip-version 4         set ike-version 1         set local-gw 0.0.0.0         set nattraversal disable         set keylife 28800         set authmethod psk         set mode main         set peertype any         set mode-cfg disable         set proposal aes256-sha1         set localid ''         set localid-type auto         set negotiate-timeout 30         set fragmentation enable         set dpd disable         set forticlient-enforcement disable         set comments "Anbindung ProSicherheit"         set npu-offload enable         set dhgrp 2         set wizard-type custom         set xauthtype disable         set mesh-selector-type disable         set remote-gw X.X.X.X         set monitor ''         set add-gw-route disable         set psksecret ENC dmFyLwUeD/jeqkdliQKsb7vC9S+Zfv/2xVEJNCVu7khE12gkEc41RcFf/rSDA92SQcmCXFnzNuBf7PrSiVYqvrjI6FeIr4AK4RHCM4sE+z2YYSxu+XyqL0Kd1T54/fq8xlKJzGtkDdTAKFGKs7MwJYQVzv/c5Xc/LX+duVK/tSDcvZHhB5m6yNmxdXOqo2nIiSkUBw==         set auto-negotiate enable    edit "P2_ProSicherh_P"         set phase1name "P1_ProSicherh-P"         set proposal aes256-sha1         set pfs enable         set dhgrp 2         set replay disable         set keepalive enable         set auto-negotiate disable         set keylife-type seconds         set encapsulation tunnel-mode         set comments ''         set protocol 0         set src-addr-type subnet         set src-port 0         set dst-addr-type subnet         set dst-port 0         set keylifeseconds 3600         set src-subnet X.X.X.X 255.255.0.0         set dst-subnet X.X.X.X 255.255.255.0     next

[ede_pfau]

It looks like the phase1 keylife period is set to 28800 s on the FGT, to 3600 s on the Juniper.

[MrSinners]

Also the FG has PFS enabeld which is not present in the Juniper config and most likely not enabled on the Fritzbox. The payload error is most likely caused by the key life time though.

[Sidlahar]

Hi ede_Pfau,

youre right, there was a configuration mistake. But correcting this does not solve the Problem. After upgrading the FritzBox the tunnel came up and everything works fine.

Thanks all for your Support!

Best regards

Sidlahar