NAT port is exhausted

hi, and welcome to the forums. This situation is quite seldom. Source of the trouble is that there are only 63K ports for outgoing NAT which all of your WAN sessions are using. Depending on the rate of new sessions (per second) I wouldn' t be surprised if at 10K active sessions your FGT might reach this limit soon. When the FGT is running out of available ports it has to deny sessions which ultimately leads to reduced throughput or connection failures. What you could do IMHO: - update your firmware to the latest in 4.2, being 4.2.15. There is a slight chance that intermediate bug fixes help in this situation. - you could reduce the TTL (time to live) for TCP sessions (or, more specific, HTTP sessions if these are the bulk of your outgoing traffic). Be warned that a shorter TTL will lead to a higher rate of new session setups, meaning higher CPU load. Session setup rate is limited by hardware ressources, anyway. On the other hand, used NAT ports are released earlier, making them available for reuse. TTL can be changed either for TCP in general, or for each protocol (HTTP, port 80). This will be in the CLI only. You can find the exact commands in the CLI Reference, depending on the major FortiOS version.

Thank you Ede! Right now i have around 10k sessions divided into 2.8k tcp sessions with the rest udp sessions. the traffic is basically web or skype and QQ video and audio traffic How do i check how much of NAT table is consumed to validate? I will try to limit timeout of certain traffic particularly the UDP ones, do you have recommendation for SKYPE traffic?

The diag sys session full-stat | grep ephemeral e.g misc info: session_count=41090 exp_count=0 clash=0 memory_tension_drop=0 ephemeral=853/57344 removeable=40230 Would give you a listing of sessions, but are you 100% sure your exhausting ephemeral ports? for 10K session and a 300A that doesn' t seem right. if those above two number are NOT equaling, than you don' t have a ephemeral exhaustion problem imho, and need to look else where/ FWIW Adjusting the TTL is short fix, but if you need more ephemeral ports, you will need to look at pools and probably splitting the source ip_range. What I typically do is use no more than a /22 per ip_nat Source. And and then tweak it after monitoring. e.g (inside) ( outside global ) 192.168.{0-3}.0/24 = public_address a.b.c.d 192.168.{4-7}.0/24 = public_address e.f.g.h 192.168.{8-11}.0/24 = public_address i.j.k.l Hope that helps and Ede is correct in that most model will be 63K or less, and you don' t get the full range from 0 thru 65535 .

hi emnoc This is my output misc info: session_count=9937 exp_count=0 clash=8014 memory_tension_drop=0 ephemeral=1391/32768 removeable=8544 doesnt look like an ephemeral problem Im not sure how to determine the source of NAT port exhaustion? Awhile ago the sessions reacher 25-27k connections but the error message continually appears even at 5 or 9k Furthermore how do i determine there is negative effect from that i.e. packets being dropped denied or timed out? I did try pools right now (pool of 5 public IP address) randomly used by all outgoing internet connections but no effect on the error messages

The router was kind of slow before until i turned off all service to interface https ping etc. You think it might be under attack? Another person suggested it could be the LAN side broadcasting, it has at least 3 subnets of the form 192.168.x.yy behind one LAN interface but im not sure how to trace this beyond get sys session

LAN side broadcasting

And to comment on broadcast, a true broadcast would be local and not in a session table. It should have a TTL = 1

Guys thank you for your replies. I finally found the cost but the question is why? It turns out the source of problem is a DNS entry primary DNS points used to point to ISP DNS and secondary to windows server for file sharing Now i recall changing the primary DNS to OPENDNS SERVER in both the NETWORK/OPTIONS and DHCP SERVER/Service settings One of the Internal subnets could sjype but could not browse. When you do NSLOOKUP it says server UNKNOWN which made me realize it was a DNS issue So i changed both entries from OPENDNS to ISP DNS then after all the PC were able to obtain new DNS setting by DHCP the message stopped! question is WHY the DNS setting would have an effect? OpenDNS is free, what would cause the NAT port to be exhausted by changing to OPENDNS? I dont understand this but at least the messages are finally gone! :)

hi emnoc There was no specifics in the log no IP no cause for error just a generic NAT port is exhausted message over and over again Here are my router stats get system performance status CPU states: 0% user 4% system 0% nice 96% idle Memory states: 26% used Average network usage: 11319 kbps in 1 minute, 12139 kbps in 10 minutes, 10791 kbps in 30 minutes Average sessions: 11120 sessions in 1 minute, 10881 sessions in 10 minutes, 10590 sessions in 30 minutes Virus caught: 0 total in 1 minute IPS attacks blocked: 0 total in 1 minute Uptime: 3 days, 21 hours, 13 minutes diag firewall statistic show getting traffic statistics... Browsing: 159874318 packets, 103933564323 bytes DNS: 366939049170042880 packets, 1722376374976512 bytes E-Mail: 219530405 packets, 23376 bytes FTP: 4200478015488 packets, 237623360618496 bytes Gaming: 2799 packets, 234982 bytes IM: 171798691840 packets, 858877495083008 bytes Newsgroups: 81345982 packets, 2486 bytes P2P: 0 packets, 0 bytes Streaming: 5828 packets, 480514 bytes TFTP: 5117227935731810304 packets, 1714876790847045641 bytes VoIP: 102645877505 packets, 4992585 bytes Generic TCP: 189906273959936 packets, 12757036416630784 bytes Generic UDP: 0 packets, 0 bytes Generic ICMP: 0 packets, 0 bytes Generic IP: 0 packets, 0 bytes I think you hit it when you said " can you disable local DNS from the firewall temporary ( e.g I had a problem on some 100A that was caused by all of the DNS lookups that it was trying todo )" but why is it doing all kind of lookups with the opendns one and not the isp one? How did you find out it was DNS when you had the problem? Very good tips on troubleshooting i will try each one how do i download the output of get sys session list? it pauses when i use it so i grep the output Have to look into cacti but i must turn on snmp right? What can i do with the syslog output? which is easier nagios or cacti? Seems like im still not maxing out the 300A at 100 users and 25K sessions during peak

Do you have a local DNS-server server maybe? As far as monitoring, I use a mix of cacti/nagios3 and sometime the opsview ( which is really nagios). Rare, but some times OpenNMS or Solarwinds. All has it pros and cons, but cacti is well documented for being opensource, easy to setup on debian/ubuntu and pretty much reliable. Only the fortigate mibs could be different so some cacti templates will have to be re-worked for querying. YMMV but the experience overall is good with cacti/nagios3. On OpenDNS, have you tried GooglePUBLIC DNS? Also do you have ipv6 enabled anywhere? The reason I' m mentioning this, I just ran into problems Google/Yahoo, these providers are IPv6 enabled, and DNS queries for any record , returns both the AAA and A resource-types. So some clients where querying and getting AAAA response and then their ipv6 was blackholed. So the client' s kept sending request and where in a big loop. Youtube was mainly effected btw. It was easy to spot by the number of DNS related sessions in the table.