QoS end to end

Just need a little validation check against something I am implementing yet feel like I am losing something as well. What I mean by that is, I have in my environment, an edge firewall (this is where my users get their primary Internet from), a core firewall (that is acting also as my core router for my private WAN and firewall for my datacenter servers), and my site firewalls. Most of my security inspection is happening at the site firewalls. Since those traffic flows have now been inspected at the site level, I had no inspection for that traffic at the core if it went to the edge, again with no more inspection at the edge (only inspection was for some segmented VLANs off the core to my datacenter servers, etc...). 

Now that I want to implement QoS/traffic shaping on some traffic (Apple and Microsoft blowing up my bandwidth), my assumption is that I am going to have to implement traffic shaping also end to end... so edge, core, and site. Am I thinking correct on this? I feel like this is correct, but I now have to "inspect" certain traffic flows on my core and edge which will slow that traffic down some, so I lose in that regard (speed and resource usage on those subsequent appliances). Thoughts?