Accept: session timeout during telnet

Hi,

I have a problem with one of our sites during telnet session , when i try to telnet to a server from the HQ branch the fortigate doesn't allow the connection ( I have already a policy from outside to inside allow telnet and ssh to the server) , looking in the logs i found the following "Accept: session timeout" , i also noticed that if i changed my IP i use to initiate the telnet ,the fortigate accept without a problem , what could be going wrong?