I've been running FortiOS 5.4 since beta 2 and unless you're doing uncommon things with your FortiGate, the 5.4GA is fine. The only bugs I've personally experienced are cosmetic.
Your top sustained throughput depends on what security profiles you apply to the data stream. If you do deep packet inspection combined with full NGFW and have data streams with file attachments... that's where things can slow down. It also depends on the number of such data streams. If you're talking home office to a small office with maybe a dozen people, the 50E/51E will probably be quite fine. Depending on usage, it could handle 2-3 dozen.
I assume you're going with the 51E for the local storage. This will let you store log files locally and have additional reporting capability that the 50E would not.
FYI, I'm now recommending clients buy a FortiAP 221C instead of a FortiWiFi because you get the 5Ghz radio (and the 2.4Ghz radio) that most newer computers and devices support... especially if you're in an area with interference from other access points.
I personally have no experience with the E-models, so can't speak to the issues being reported.