Troubleshooting IPsec-VPN connection attempts

[#1]

Hello all, I am a new to fortigate and I have came into a dead end in my attempts to establish a successful ipsec vpn connection. I would really appreciate any help. On the fortigate unit an ipsec connection is configured as interface mode dialup-server, with certificate based authentication. From the client side forticlient is used with proper certificates installed and matched configuration. This is the output of the connection test: Now install tunnels into kernel: 1 sys_get_local_gwy() called: [in] remote gw: (gateway ip). [in] Next hop: 0.0.0.0 sys_get_local_gwy() called: [in] remote gw: (gateway ip). [in] Next hop: 0.0.0.0 sys_get_local_gwy() called: [in] remote gw: (gateway ip). [in] Next hop: 0.0.0.0 sys_get_local_gwy() called: [in] remote gw: (gateway ip). [in] Next hop: 0.0.0.0 End installing tunnels Got a kernel message Detect local gateway for peer: (gateway ip) sys_get_local_gwy() called: [in] remote gw: (gateway ip). [in] Next hop: 0.0.0.0 sys_get_local_gwy() called: [in] remote gw: (gateway ip). [in] Next hop: 10.0.0.138 Get sa_connect message...10.0.0.78->(gateway ip):500, natt_mode=0 Using new connection...natt_mode=0 Set connection name = (connection name). Tunnel 10.0.0.78 ---> (gateway ip):500,natt_en=1 is starting negotiation Will negotiate a normal SA (client' s certificate name) cert (client' s certificate name) found set_phase1_id(): RSA -> ID_DER_ASN1_DN and it holds on that last line.. This is the log messages from the fortigate unit: Message meets Alert condition date=2010-10-11 time=12:47:27 devname=(devname) device_id=(device_id) log_id=0101037128 type=event subtype=ipsec pri=error fwver=040004 vd=" root" msg=" progress IPsec phase 1" action=" negotiate" rem_ip=(client' s public ip) loc_ip=(gateway ip) rem_port=885 loc_port=500 out_intf=" wan2" cookies=" 23585904a0094a6a/0000000000000000" user=" N/A" group=" N/A" xauth_user=" N/A" xauth_group=" N/A" vpn_tunnel=" IPSec-VPN" status=failure init=remote mode=main dir=inbound stage=1 role=responder result=ERROR Message meets Alert condition date=2010-10-11 time=12:47:27 devname=(devname) device_id=F(device_id) log_id=0101037124 type=event subtype=ipsec pri=error fwver=040004 vd=" root" msg=" IPsec phase 1 error" action=" negotiate" rem_ip=(client' s public ip) loc_ip=(gateway ip) rem_port=885 loc_port=500 out_intf=" wan2" cookies=" 23585904a0094a6a/0000000000000000" user=" N/A" group=" N/A" xauth_user=" N/A" xauth_group=" N/A" vpn_tunnel=" IPSec-VPN" status=negotiate_error error_reason=peer SA proposal not match local policy peer_notif=N/A Any ideas for the possible reasons for this? How can I further, more granularly troubleshoot this? I know there are diagnose cli commands but they are not in the cli guide and from what I did try - I couldn' t get any output.

[ede_pfau]

Hi, and welcome to the forums! The VPN can' t negotiate a phase 1 SA, probably because the cert doesn' t fit. Can you change the authentication to PSK just to make sure the rest of the parameters are OK?

[#3]

Thanks ede_pfau This is the output from the connection test using PSK: Now install tunnels into kernel: 1 sys_get_local_gwy() called: [in] remote gw: (gateway ip). [in] Next hop: 0.0.0.0 sys_get_local_gwy() called: [in] remote gw: (gateway ip). [in] Next hop: 0.0.0.0 sys_get_local_gwy() called: [in] remote gw: (gateway ip). [in] Next hop: 0.0.0.0 sys_get_local_gwy() called: [in] remote gw: (gateway ip). [in] Next hop: 0.0.0.0 End installing tunnels Got a kernel message Detect local gateway for peer: (gateway ip) sys_get_local_gwy() called: [in] remote gw: (gateway ip). [in] Next hop: 0.0.0.0 sys_get_local_gwy() called: [in] remote gw: (gateway ip). [in] Next hop: 10.0.0.138 Get sa_connect message...10.0.0.78->(gateway ip):500, natt_mode=0 Using new connection...natt_mode=0 Set connection name = (connection name). Tunnel 10.0.0.78 ---> (gateway ip):500,natt_en=1 is starting negotiation Will negotiate a normal SA Got a kernel message Detect local gateway for peer: (gateway ip) sys_get_local_gwy() called: [in] remote gw: (gateway ip). [in] Next hop: 0.0.0.0 sys_get_local_gwy() called: [in] remote gw: (gateway ip). [in] Next hop: 10.0.0.138 Get sa_connect message...10.0.0.78->(gateway ip):500, natt_mode=0 Using old connection...natt_mode=0 Create new connection for new phase1. Set connection name = (connection name). Tunnel 10.0.0.78 ---> (gateway ip):500,natt_en=1 is starting negotiation Will negotiate a normal SA Got a kernel message Detect local gateway for peer: (gateway ip) sys_get_local_gwy() called: [in] remote gw: (gateway ip). [in] Next hop: 0.0.0.0 sys_get_local_gwy() called: [in] remote gw: (gateway ip). [in] Next hop: 10.0.0.138 Get sa_connect message...10.0.0.78->(gateway ip):500, natt_mode=0 Using old connection...natt_mode=0 Create new connection for new phase1. Set connection name = (connection name). Tunnel 10.0.0.78 ---> (gateway ip):500,natt_en=1 is starting negotiation Will negotiate a normal SA and repeats.. The log Messages from the fortigate unit are the same as before..

[ede_pfau]

so the basic negotiations fail. Please post the phase1 and phase2 definitions, along with both subnets involved (net+mask). From the CLI: get vpn ipsec phase1-interface get vpn ipsec phase2-interface ...if you are using interface based VPN (which I strongly recommend), and get system interface physical for the FG, and ipconfig /all for the FC side.

[#5]

Phase 1 definitions: name - IPSec-VPN remote gateway - dialup user local interface - wan2 mode - main authentication method - preshared key peer options - accept any peer id Enable ipsec interface mode ike verision - 1 local gateway ip - main interface ip encryption - aes256 authentication - sha1 dh group - 2 keylife - 28800 nat traversl - enabled keepalive frequency - 10 dead peer detection - enabled Phase 2 definitions: name - IPSec-VPN-P2 phase 1 - IPSec-VPN encryption - aes128 authentication - sha1 enable replay detection enable PFS dh group - 2 keylife - 1800 seconds autokey keep alive - enabled quick mode selector - all zeroes subnets: 10.0.0.0/8 - on both sides FW # get vpn ipsec phase1-interface == [ IPSec-VPN ] name: IPSec-VPN == [ Home ] name: Home FW # get vpn ipsec phase2-interface == [ IPSec-VPN-P2 ] name: IPSec-VPN-P2 FW # get system interface physical == [onboard] ==[dmz1] mode: static ip: 10.202.192.107 255.255.255.248 ipv6: ::/0 status: up speed: 100Mbps (Duplex: full) ==[dmz2] mode: static ip: 0.0.0.0 0.0.0.0 ipv6: ::/0 status: down speed: n/a ==[internal] mode: static ip: 10.0.0.254 255.128.0.0 ipv6: ::/0 status: up speed: 10Mbps (Duplex: half) ==[wan1] mode: pppoe ip: 0.0.0.0 0.0.0.0 ipv6: ::/0 status: down --More-- speed: n/a --More-- ==[wan2] mode: pppoe ip: x.x.x.x 255.255.255.255 ipv6: ::/0 status: up speed: 100Mbps (Duplex: full) ==[modem] mode: static ip: 0.0.0.0 0.0.0.0 ipv6: ::/0 status: down speed: n/a ipconfig /all: Windows IP Configuration Host Name . . . . . . . . . . . . : Laptop Primary Dns Suffix . . . . . . . : x.local Node Type . . . . . . . . . . . . : Hybrid IP Routing Enabled. . . . . . . . : No WINS Proxy Enabled. . . . . . . . : No DNS Suffix Search List. . . . . . : x.local x.local Ethernet adapter Wireless Network Connection: Media State . . . . . . . . . . . : Media disconnected Description . . . . . . . . . . . : Intel(R) WiFi Link 5100 AGN Physical Address. . . . . . . . . : 00-22-FB-9A-26-30 Ethernet adapter Local Area Connection: Connection-specific DNS Suffix . : x.local Description . . . . . . . . . . . : Intel(R) 82567LM Gigabit Network Con nection Physical Address. . . . . . . . . : 00-24-E8-A8-EC-83 Dhcp Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes IP Address. . . . . . . . . . . . : 10.0.0.78 Subnet Mask . . . . . . . . . . . : 255.0.0.0 Default Gateway . . . . . . . . . : 10.0.0.138 DHCP Server . . . . . . . . . . . : 10.0.0.10 DNS Servers . . . . . . . . . . . : 10.0.0.10 Primary WINS Server . . . . . . . : 10.0.0.10 Lease Obtained. . . . . . . . . . : Monday, October 11, 2010 10:23:19 AM Lease Expires . . . . . . . . . . : Friday, October 15, 2010 10:23:19 AM

[ede_pfau]

subnets: 10.0.0.0/8 - on both sides

whoa. Source and target networks should have distinct IP ranges. You could make your life easier if you wouldn' t use so enormous ranges - 10.0.0.0/9 behind the Fortigate, 10.0.0.0/8 on the client. That is more than 8 million IP addresses overlap. Frankly, it might just be too early to debug this configuration. My impression is that you haven' t yet read the VPN guide thoroughly (fortigate-ipsec-40-mr2.pdf on docs.fortinet.com). There they give examples of the various VPN scenarios, esp. the " FortiClient dialup-client configuration" and with that you won' t have any problem. You can expand the example later when it' s working e.g. with certificates and/or a user database.

[#7]

Thanks for the help ede As I understand it, as long as I use a VIP which isn' t used on the remote network -it sould be ok, am I wrong? Anyway, I have changed my configuration to follow the example in the guide - in forticlient - I had set the " remote network" setting to a subnet different than the VIP and I have changed the firewall policies accordingly. Still I cannot connect and I get the same error messages. Any idea what could be wrong?

[ede_pfau]

You have 2 obstacles to clear: - local and remote network cannot overlap - how should a packet for a remote host determine it is not finding its destination on the local network? - the VPN parameters don' t match. If you really insist on overlapping subnets (if you have to) the VIP might be a way out. But how do you think that the routing on the laptop can work if you specify a local IP as the destination? Have you worked through the VPN Guide I cited? They deliberately don' t use the same subnet on both sides of a VPN.

[#9]

I have changed the encryption method in the phase 1 policy on the fortigate unit to AES128 (and accordingly on the client) and it solved the problem.. Is it a known issue? perhaps my specific client machine has problems with AES256, I didn' t made connection attempts on another machines. Anyway, just FYI and thanks for the help

[rwpatterson]

There is (was?) a problem with AES256 on some versions of firmware. Not sure which ones, but it' s out there in the forums... You never specified your version, by the way.