Working example is set-tos needed

Based on my research so far, looks like they're trying to consolidate/unify terminology/syntax related to TOS/DSCP to TOS only. ex) They changed the syntax for firewall policy/SD-WAN rule from dscp-match/dscp-value/dscp-negate to tos-mask/tos/tos-negate with 6.2. There is no explanation at Traffic Shaping section of 6.2 cookbook (no handbook available yet) but at SD-WAN/DSCP matching section

https://docs.fortinet.com/document/fortigate/6.2.0/new-features/159007/dscp-matching-shaping

So my assumption in the previous post seems to be correct and when you need to match/handle DSCP bits (upper 6bits), you need to mask the TOS byte (8bits) and use [DSCP value x 4] to match what you're looking for. I have a ticket opened related to this at TAC. So I'll verify this and update this thread.

It's simply reiterating how bitwise operation would work. Exactly in the same way with subnet-mask works. For DSCP, you need to match the top 6 bits. So in binary the mask need to be "11111100" (=0xfc).

The main question is what you want to do (how you want to queue) with the matching. EF and AF43 packets into Queue0? The rest to be in Queue 2? Without setting any guaranteed bandwidth to other types of traffic? Then I wouldn't use shaping-policy, but do everything at ingress point. I'm assuming you don't have multiple vdoms.

  config sys global      set traffic-priority dscp      set traffic-priority-level  low (=2) <- default is high (=0). you need to change it.   end   config sys dscp-based-priority     edit 46       set ds 46

      set priority high (=0) <- default value     next

    edit 38

      set ds 38

      set priority high (=0)

   next end

My problem with tos/tos-mask not working as expected was because I didn't disable SIP session-helper/ALG. Now it's working as configured (It added us another reason not to use the session-helper/ALG).

My original config was confirmed. The tos/tos-mask in shaping-policy is 8 bit TOS Byte, not 4 bit TOS filed. So if you're matching with DSCP codes, tos-mask needs to be 0xfc(11111100). Then tos can be 0xb8 (EF for RTP), 0x68 (AF31 for SIP), or other values.

However, you need to be aware of FGT's QoS operation; initial prioritization at ingress, 0 - 2, 0(high) is the default, which you can change per incoming DSCP code under "config sys dscp-based-priority" after specifying "config sys global/set traffic-priority dscp", and policy-based (with shaping-policy+shaper) priority adjustment, +1 - +2.

So if you just want to put specific traffic like EF and AF31 to the highest priority queue (0) only based on incoming DSCP codes while putting the rest into a lower queue, you should do it at the global dscp-based-priority. Because that's the only way to put them in queue0. If you use shaping-policies instead, the highest is queue1 (initial '0' +1). But that method is still valid because you can keep the relative priorities between traffic types (DSCP codes) if you match 'tos' and set priorities properly.

I am using shaping profiles (type queueing) with max, guaranteed bw and priorities. Profiles are consisted of shaping policies that should match dscp marked packets using tos/tos mask. I'm not using any other criteria here. Profiles are applied on interfaces using "set egress-shaping-profile".

Whlie using cmd "diagnose netlink intf-class list" i can see some classes matching while others don't. So, packets marked with af31 match the policy with a tos of 0x68 and mask 0xfc which increments counter "Sent packets" using the above cmd. Packets marked EF with tos 0xb8 and mask 0xfc do not increment the above counter.

Im thinking on reverting this from dscp matching to plain source destination port match which seems to work although all tests are not conducted in a congestive state.

PS.  Global setting is on DSCP values 

You are right. I am on 6.4 and by the looks of it and the TAC person, this may be a bug that seems to be triggered when the traffic matching the shaping profile is passed through a policy that has hardware acceleration on. When you disable acceleration traffic matches on shaping profile.