Error running large script

[rafaesp]

I'm trying to develop a script to create a large amount of firewall policies but I'm getting the error "Broken Pipe" when the script is larger than ~260kb. To send the script, I'm using a library available with my programming language to open a ssh connection and send the commands but the connection is broken immediately when the command is larger than those ~260kb.

I'm testing against FortiWiFi-60D v5.2.3.

Any idea of what may be happening? Thanks a lot.

[emnoc]

Are you sure it's the script or  the size? If you cut the script down in half and execute partA and then partB does it work? Do you have compression enabled ? ( if yes try turning it off ) and  the same for the option TcpKA? ( enable it or disable it and see if the output changes )

I never seen scripts that big,  but have create  in excess of  3K objects and addrgrp and never had errors using bash or piping thru ssh  and various ssh clients seems to be better or less  ( i.e OpenSSH )

[ede_pfau]

Seriously, thousands of policies on a FG-60D??

Have you checked that against the Maximum Values Matrix, for instance here http://help.fortinet.com/fgt/56/max-values/5-6-3/max-values.html ?

OK it says 5.000 policies hard limit but I guess you'll run into memory issues far below that.

[rwpatterson]

The Max values matrix referenced is for a different firmware level. May be different on 5.2.x.

[ede_pfau]

might have been but isn't...

[emnoc]

So I would split the  script in  half run 2 batch per-se, that  might expose the issue(s). If the script in batch1 runs 100% and then batch2, fails it could be limits issues.

As far as max-value, system log should show a message if max values are exceed. So he could run thru that and see what if any messages.

[rafaesp]

Thanks for your responses. Cutting the script in half works ok.

I'm not hitting any max because I'm testing with a script that only edits one rule and sets a comment, but thousands of times. I think it demonstrates that adding more lines to the script until it reach certain size causes up the issue. I'm now using this library: https://github.com/spotify/pyfg/tree/master/pyFG And my code looks like this:

#!/usr/bin/env python

from pyFG import FortiOS


if __name__ == '__main__':
    f = open('large_script.txt', 'r')
    candidate = f.read()
    f.close()

    d = FortiOS("YOUR_HOST", vdom="YOUR_VDOM", username="YOUR_USER", password="YOUT_PASSWD")
    d.open()

    print d.commit(candidate)

    d.close()

The large file I'm using hits the max size of the forum but I upload one where you can duplicate the lines that sets the comments thousand of times until the size of the file is more than 260kb.

Thanks again.