This sounds similar to a bug we had when trying to install a package update to our firewalls.
The problem occurred when we removed a per-ip shaper from a policy, the FMG would seg fault when doing an installation preview. This was between 5.0.7 (FMG) & 5.0.7 (FGT)
My workaround was to manually make those changes on the actual Fortigates so that they matched the package on the FMG, then I was able to do the install.
This was fixed in FMG 5.2.0, or at least the release notes say it' s been fixed.
My best advice would be to upgrade the FMG to 5.2.0, it' s backwards compatible to 5.0.5 through 5.0.9 so you should be fine.
While you' re issue may well be different to mine, the workaround may still work, ie; manually update the Fortigate to match your package and see if it installs.
Regards,
Matthew