Turns out, it was the FortiGate that was to blame for this one. Spectrum is off the hook!
After battling this issue for months - I called into support and opened a new case for this issue (I believe that makes three times I have done this). Got someone new from the UK looking at the issue, and the guy was a real whiz. In the middle of explaining the issue, he knew exactly what the problem was.
The gist of the issue is this: when FortiGate is sending packets via a VPN tunnel, and the tunnel is presently inaccessible (for that very brief moment in time) it will go fishing for a new route. Since the tunnel is inaccessible many times a day (cable modem glitches, it's rebuilding after a timeout, etc) there are random clients who end up with the FortiGate "browsing for a new route". It's go to route when this happens, is the WAN interface (0.0.0.0/0). The cable modem is probably making the connection that private range IP's shouldn't be headed to the Internet on their public IP address. So it's getting in the way and eating packets.
The solution was very easy. It's called "blackhole routing". So you go Network > Static Routes. In there you will see some routes where the destination is your VPN taffic, and the Interface is your VPN tunnel. You simply add a new additional route where the destination is your VPN traffic, but the Interface is "Blackhole" (it's in the dropdown list). Then you set the Distance to something far out and unreachable (the tech set it to 250). FIXED!
Now if the client should happen to try and hit a host at the worst possible time, the traffic dies. The retry that is occurring in the background gets things going again, and the client never skips a beat.
Not sure why the first few techs didn't recognize this issue. But MEGA GOOD FEEDBACK is in order to the guy that caught it. ;)