High CPU in AWS Fortigate t2-small instnace

[Mohammed-Mustafa]

 have Fortigate OS version 7.2.2 running on AWS using t2.small instance, it periodically has a high CPU, this ccurs at time there's no actual load on the firewall, how to slove this???

I have the following log: 

Description:

########## script name: autod.57 ########## ========== #1, 2022-11-02 03:50:07 ========== FGT $ diagnose debug cli 8 Debug messages will be on for 30 minutes. FGT $ diagnose debug console timestamp enable FGT $ diagnose debug enable FGT $ diagnose debug crashlog read 2: 2022-08-10 17:59:49 (Release) 3: 2022-08-10 17:59:49 <21108> application wad 4: 2022-08-10 17:59:49 <21108> *** signal 11 (Segmentation fault) received *** 5: 2022-08-10 17:59:49 <21108> AVDB 90.04943(08/10/0022 07:26) 6: 2022-08-10 17:59:49 <21108> ETDB 90.04943(08/10/0022 07:26) 7: 2022-08-10 17:59:49 <21108> AVSO 04000000AVEN00701062732202160208 8: 2022-08-10 17:59:49 <21108> Register dump: 9: 2022-08-10 17:59:49 <21108> RAX: fffffffffffffffc RBX: 000000000000000f 10: 2022-08-10 17:59:49 <21108> RCX: ffffffffffffffff RDX: 0000000000000039 11: 2022-08-10 17:59:49 <21108> R08: 000000000000037a R09: 0000000000000009 12: 2022-08-10 17:59:49 <21108> R10: 000000000000037a R11: 0000000000000246 13: 2022-08-10 17:59:49 <21108> R12: 0000000000000001 R13: 0000000000000002 14: 2022-08-10 17:59:49 <21108> R14: 0000000000000002 R15: 0000000000000003 15: 2022-08-10 17:59:49 <21108> RSI: 00000000104bf7d0 RDI: 000000000000000a 16: 2022-08-10 17:59:49 <21108> RBP: 00007fff815dc220 RSP: 00007fff815dc208 17: 2022-08-10 17:59:49 <21108> RIP: 00007f081cba3016 EFLAGS: 0000000000000246 18: 2022-08-10 17:59:49 <21108> CS: 0033 FS: 0000 GS: 0000 19: 2022-08-10 17:59:49 <21108> Trap: 0000000000000000 Error: 0000000000000000 20: 2022-08-10 17:59:49 <21108> OldMask: 0000000000000000 21: 2022-08-10 17:59:49 <21108> CR2: 0000000000000000 22: 2022-08-10 17:59:49 <21108> stack: 0x7fff815dc208 - 0x7fff815dc450 23: 2022-08-10 17:59:49 <21108> Backtrace: 24: 2022-08-10 17:59:49 <21108> [0x7f081cba3016] => /usr/lib/x86_64-linux-gnu/libc.so.6 25: 2022-08-10 17:59:49 (epoll_wait+0x00000016) liboffset 000fa016 26: 2022-08-10 17:59:49 <21108> [0x01b970ea] => /bin/wad 27: 2022-08-10 17:59:49 <21108> [0x01c37416] => /bin/wad 28: 2022-08-10 17:59:49 <21108> [0x0044c767] => /bin/wad 29: 2022-08-10 17:59:49 <21108> [0x7f081caccdeb] => /usr/lib/x86_64-linux-gnu/libc.so.6 30: 2022-08-10 17:59:49 (__libc_start_main+0x000000eb) liboffset 00023deb 31: 2022-08-10 17:59:49 <21108> [0x00447dba] => /bin/wad 32: 2022-08-10 17:59:49 <21108> fortidev 6.0.1.0005 33: 2022-08-10 17:59:49 <21108> process=wad type=2 idx=0 av-scanning=no total=2010 free=324 34: 2022-08-10 17:59:49 mmu=2530767 mu=2502011 m=52957 f=23129 r=0 35: 2022-08-10 17:59:49 <21108> cur_bank=(nil) curl_tl=0xbf731e0 curl_tm=0x7f07fe0ed048 36: 2022-08-10 17:59:49 <21108> (session info) 37: 2022-08-10 17:59:49 [AV Engine <21108>] AV Engine version: 6.4.273 38: 2022-08-10 17:59:49 [AV Engine <21108>] Last file info: 39: 2022-08-10 17:59:49 [AV Engine <21108>] filename: , filesize: 0, filebuffer: (nil) 40: 2022-08-10 17:59:49 [AV Engine <21108>] Native script imagebase: 0x7f07fd411000 41: 2022-08-10 17:59:49 [AV Engine <21108>] Native script imagesize: 0x8000 42: 2022-08-10 17:59:49 [AV Engine <21108>] AV Engine imagebase: 0x7f07fe11e000 43: 2022-08-10 18:59:53 wad crashed 1 times. The latest crash was at 2022-08-10 17:59:49. 44: 2022-08-11 12:25:04 the killed daemon is /bin/wad: status=0x0 45: 2022-08-11 18:25:05 the killed daemon is /bin/wad: status=0x0 46: 2022-08-12 00:25:05 the killed daemon is /bin/wad: status=0x0 47: 2022-08-12 06:25:06 the killed daemon is /bin/wad: status=0x0 48: 2022-08-12 12:25:07 the killed daemon is /bin/wad: status=0x0 49: 2022-08-12 18:25:07 the killed daemon is /bin/wad: status=0x0 50: 2022-08-13 00:25:07 the killed daemon is /bin/wad: status=0x0 51: 2022-08-13 06:25:08 the killed daemon is /bin/wad: status=0x0 52: 2022-08-13 12:25:09 the killed daemon is /bin/wad: status=0x0 53: 2022-08-13 18:25:10 the killed daemon is /bin/wad: status=0x0 54: 2022-08-14 00:25:11 the killed daemon is /bin/wad: status=0x0 55: 2022-08-14 06:25:11 the killed daemon is /bin/wad: status=0x0 56: 2022-08-14 12:25:12 the killed daemon is /bin/wad: status=0x0 57: 2022-08-14 18:25:12 the killed daemon is /bin/wad: status=0x0 58: 2022-08-15 00:25:13 the killed daemon is /bin/wad: status=0x0 59: 2022-08-15 06:25:14 the killed daemon is /bin/wad: status=0x0 60: 2022-08-15 12:25:14 the killed daemon is /bin/wad: status=0x0 61: 2022-08-15 18:25:14 the killed daemon is /bin/wad: status=0x0 62: 2022-08-16 00:25:15 the killed daemon is /bin/wad: status=0x0 63: 2022-08-16 06:25:16 the killed daemon is /bin/wad: status=0x0 64: 2022-08-16 12:25:16 the killed daemon is /bin/wad: status=0x0 65: 2022-08-16 18:25:17 the killed daemon is /bin/wad: status=0x0 66: 2022-08-17 00:25:18 the killed daemon is /bin/wad: status=0x0 67: 2022-08-17 02:51:20 the killed daemon is /bin/csfd: status=0x0 68: 2022-08-17 02:51:21 the killed daemon is /bin/eap_proxy: status=0x0 69: 2022-08-17 06:25:18 the killed daemon is /bin/wad: status=0x0 70: 2022-08-17 12:25:19 the killed daemon is /bin/wad: status=0x0 71: 2022-08-17 18:25:20 the killed daemon is /bin/wad: status=0x0 72: 2022-08-18 00:25:20 the killed daemon is /bin/wad: status=0x0 73: 2022-08-18 06:25:21 the killed daemon is /bin/wad: status=0x0 74: 2022-08-18 12:25:21 the killed daemon is /bin/wad: status=0x0 75: 2022-08-18 18:25:22 the killed daemon is /bin/wad: status=0x0 76: 2022-08-19 00:25:22 the killed daemon is /bin/wad: status=0x0 77: 2022-08-19 06:25:23 the killed daemon is /bin/wad: status=0x0 78: 2022-08-19 12:25:24 the killed daemon is /bin/wad: status=0x0 79: 2022-08-19 18:25:24 the killed daemon is /bin/wad: status=0x0 80: 2022-08-20 00:25:24 the killed daemon is /bin/wad: status=0x0 81: 2022-08-20 06:25:25 the killed daemon is /bin/wad: status=0x0 82: 2022-08-20 12:25:25 the killed daemon is /bin/wad: status=0x0 83: 2022-08-20 18:25:26 the killed daemon is /bin/wad: status=0x0 84: 2022-08-21 00:25:27 the killed daemon is /bin/wad: status=0x0 85: 2022-08-21 06:25:27 the killed daemon is /bin/wad: status=0x0 86: 2022-08-21 12:25:28 the killed daemon is /bin/wad: status=0x0 87: 2022-08-21 18:25:28 the killed daemon is /bin/wad: status=0x0 88: 2022-08-22 00:25:29 the killed daemon is /bin/wad: status=0x0 89: 2022-08-22 06:25:30 the killed daemon is /bin/wad: status=0x0 90: 2022-08-22 12:25:31 the killed daemon is /bin/wad: status=0x0 91: 2022-08-22 18:25:31 the killed daemon is /bin/wad: status=0x0 92: 2022-08-23 00:25:32 the killed daemon is /bin/wad: status=0x0 93: 2022-08-23 06:25:32 the killed daemon is /bin/wad: status=0x0 94: 2022-08-23 12:25:32 the killed daemon is /bin/wad: status=0x0 95: 2022-08-23 18:25:33 the killed daemon is /bin/wad: status=0x0 96: 2022-08-24 00:25:33 the killed daemon is /bin/wad: status=0x0 97: 2022-08-24 06:25:34 the killed daemon is /bin/wad: status=0x0 98: 2022-08-24 12:25:34 the killed daemon is /bin/wad: status=0x0 99: 2022-08-24 18:25:34 the killed daemon is /bin/wad: status=0x0 100: 2022-08-25 00:25:35 the killed daemon is /bin/wad: status=0x0 101: 2022-08-25 06:25:35 the killed daemon is /bin/wad: status=0x0 102: 2022-08-25 12:25:36 the killed daemon is /bin/wad: status=0x0 103: 2022-08-25 18:25:37 the killed daemon is /bin/wad: status=0x0 104: 2022-08-26 00:25:38 the killed daemon is /bin/wad: status=0x0 105: 2022-08-26 06:25:38 the killed daemon is /bin/wad: status=0x0 106: 2022-08-26 12:25:39 the killed daemon is /bin/wad: status=0x0 107: 2022-08-26 18:25:39 the killed daemon is /bin/wad: status=0x0 108: 2022-08-27 00:25:39 the killed daemon is /bin/wad: status=0x0 109: 2022-08-27 06:25:40 the killed daemon is /bin/wad: status=0x0 110: 2022-08-27 12:25:40 the killed daemon is /bin/wad: status=0x0 111: 2022-08-27 18:25:40 the killed daemon is /bin/wad: status=0x0 112: 2022-08-28 00:25:41 the killed daemon is /bin/wad: status=0x0 113: 2022-08-28 06:25:42 the killed daemon is /bin/wad: status=0x0 114: 2022-08-28 12:25:43 the killed daemon is /bin/wad: status=0x0 115: 2022-08-28 18:25:43 the killed daemon is /bin/wad: status=0x0 116: 2022-08-29 00:25:43 the killed daemon is /bin/wad: status=0x0 117: 2022-08-29 06:25:44 the killed daemon is /bin/wad: status=0x0 118: 2022-08-29 12:25:45 the killed daemon is /bin/wad: status=0x0 119: 2022-08-29 18:25:45 the killed daemon is /bin/wad: status=0x0 120: 2022-08-30 00:25:46 the killed daemon is /bin/wad: status=0x0 121: 2022-08-30 06:25:47 the killed daemon is /bin/wad: status=0x0 122: 2022-08-30 12:25:47 the killed daemon is /bin/wad: status=0x0 123: 2022-08-30 18:25:47 the killed daemon is /bin/wad: status=0x0 124: 2022-08-31 00:25:48 the killed daemon is /bin/wad: status=0x0 125: 2022-08-31 06:25:49 the killed daemon is /bin/wad: status=0x0 126: 2022-08-31 12:25:50 the killed daemon is /bin/wad: status=0x0 127: 2022-08-31 18:25:50 the killed daemon is /bin/wad: status=0x0 128: 2022-09-01 00:25:51 the killed daemon is /bin/wad: status=0x0 129: 2022-09-01 06:25:51 the killed daemon is /bin/wad: status=0x0 130: 2022-09-01 12:25:51 the killed daemon is /bin/wad: status=0x0 131: 2022-09-01 18:25:52 the killed daemon is /bin/wad: status=0x0 132: 2022-09-02 00:25:52 the killed daemon is /bin/wad: status=0x0 133: 2022-09-02 04:10:03 service=kernel conserve=on total="2010 MB" used="1776 MB" red="1769 MB" 134: 2022-09-02 04:10:03 green="1648 MB" msg="Kernel enters memory conserve mode" 135: 2022-09-02 04:10:04 MemTotal: 2058868 kB 136: 2022-09-02 04:10:04 MemFree: 103836 kB 137: 2022-09-02 04:10:04 Buffers: 168 kB 138: 2022-09-02 04:10:04 Cached: 1119616 kB 139: 2022-09-02 04:10:04 SwapCached: 0 kB 140: 2022-09-02 04:10:04 Active: 820948 kB 141: 2022-09-02 04:10:04 Inactive: 579992 kB 142: 2022-09-02 04:10:04 Active(anon): 820932 kB 143: 2022-09-02 04:10:04 Inactive(anon): 579572 kB 144: 2022-09-02 04:10:04 Active(file): 16 kB 145: 2022-09-02 04:10:04 Inactive(file): 420 kB 146: 2022-09-02 04:10:04 Unevictable: 195476 kB 147: 2022-09-02 04:10:04 Mlocked: 0 kB 148: 2022-09-02 04:10:04 SwapTotal: 0 kB 149: 2022-09-02 04:10:04 SwapFree: 0 kB 150: 2022-09-02 04:10:04 Dirty: 20 kB 151: 2022-09-02 04:10:04 Writeback: 36 kB 152: 2022-09-02 04:10:04 AnonPages: 476592 kB 153: 2022-09-02 04:10:04 Mapped: 195952 kB 154: 2022-09-02 04:10:04 Shmem: 923876 kB 155: 2022-09-02 04:10:04 Slab: 111732 kB 156: 2022-09-02 04:10:04 SReclaimable: 21064 kB 157: 2022-09-02 04:10:04 SUnreclaim: 90668 kB 158: 2022-09-02 04:10:04 KernelStack: 2080 kB 159: 2022-09-02 04:10:04 PageTables: 31940 kB 160: 2022-09-02 04:10:04 NFS_Unstable: 0 kB 161: 2022-09-02 04:10:04 Bounce: 0 kB 162: 2022-09-02 04:10:04 WritebackTmp: 0 kB 163: 2022-09-02 04:10:04 CommitLimit: 1029432 kB 164: 2022-09-02 04:10:04 Committed_AS: 19133880 kB 165: 2022-09-02 04:10:04 VmallocTotal: 34359738367 kB 166: 2022-09-02 04:10:04 VmallocUsed: 138940 kB 167: 2022-09-02 04:10:04 VmallocChunk: 34359528915 kB 168: 2022-09-02 04:10:04 HugePages_Total: 0 169: 2022-09-02 04:10:04 HugePages_Free: 0 170: 2022-09-02 04:10:04 HugePages_Rsvd: 0 171: 2022-09-02 04:10:04 HugePages_Surp: 0 172: 2022-09-02 04:10:04 Hugepagesize: 2048 kB 173: 2022-09-02 04:10:04 DirectMap4k: 4096 kB 174: 2022-09-02 04:10:04 DirectMap2M: 2093056 kB 175: 2022-09-02 04:10:32 service=kernel conserve=exit total="2010 MB" used="1555 MB" red="1769 MB" 176: 2022-09-02 04:10:32 green="1648 MB" msg="Kernel exits memory conserve mode" 177: 2022-09-02 06:25:52 the killed daemon is /bin/wad: status=0x0 178: 2022-09-02 12:25:53 the killed daemon is /bin/wad: status=0x0 179: 2022-09-02 18:25:54 the killed daemon is /bin/wad: status=0x0 180: 2022-09-03 00:25:54 the killed daemon is /bin/wad: status=0x0 181: 2022-09-03 06:25:55 the killed daemon is /bin/wad: status=0x0 182: 2022-09-03 12:25:55 the killed daemon is /bin/wad: status=0x0 183: 2022-09-03 18:25:55 the killed daemon is /bin/wad: status=0x0 184: 2022-09-04 00:25:56 the killed daemon is /bin/wad: status=0x0 185: 2022-09-04 06:25:57 the killed daemon is /bin/wad: status=0x0 186: 2022-09-04 12:25:58 the killed daemon is /bin/wad: status=0x0 187: 2022-09-04 18:25:58 the killed daemon is /bin/wad: status=0x0 188: 2022-09-05 00:25:59 the killed daemon is /bin/wad: status=0x0 189: 2022-09-05 06:26:00 the killed daemon is /bin/wad: status=0x0 190: 2022-09-05 12:26:00 the killed daemon is /bin/wad: status=0x0 191: 2022-09-05 18:26:01 the killed daemon is /bin/wad: status=0x0 192: 2022-09-06 00:26:01 the killed daemon is /bin/wad: status=0x0 193: 2022-09-06 06:26:02 the killed daemon is /bin/wad: status=0x0 194: 2022-09-06 12:26:02 the killed daemon is /bin/wad: status=0x0 195: 2022-09-06 18:26:03 the killed daemon is /bin/wad: status=0x0 196: 2022-09-07 00:26:03 the killed daemon is /bin/wad: status=0x0 197: 2022-09-07 06:26:03 the killed daemon is /bin/wad: status=0x0 198: 2022-09-07 12:26:03 the killed daemon is /bin/wad: status=0x0 199: 2022-09-07 18:26:04 the killed daemon is /bin/wad: status=0x0 200: 2022-09-08 00:26:05 the killed daemon is /bin/wad: status=0x0 201: 2022-09-08 06:26:05 the killed daemon is /bin/wad: status=0x0 202: 2022-09-08 12:26:06 the killed daemon is /bin/wad: status=0x0 203: 2022-09-08 18:26:06 the killed daemon is /bin/wad: status=0x0 204: 2022-09-09 00:26:07 the killed daemon is /bin/wad: status=0x0 205: 2022-09-09 06:26:07 the killed daemon is /bin/wad: status=0x0 206: 2022-09-09 12:26:07 the killed daemon is /bin/wad: status=0x0 207: 2022-09-09 18:26:08 the killed daemon is /bin/wad: status=0x0 208: 2022-09-10 00:26:08 the killed daemon is /bin/wad: status=0x0 209: 2022-09-10 06:26:09 the killed daemon is /bin/wad: status=0x0 210: 2022-09-10 12:26:10 the killed daemon is /bin/wad: status=0x0 211: 2022-09-10 18:26:11 the killed daemon is /bin/wad: status=0x0 212: 2022-09-11 00:26:11 the killed daemon is /bin/wad: status=0x0 213: 2022-09-11 06:26:11 the killed daemon is /bin/wad: status=0x0 214: 2022-09-11 12:26:12 the killed daemon is /bin/wad: status=0x0 215: 2022-09-11 18:26:12 the killed daemon is /bin/wad: status=0x0 216: 2022-09-12 00:26:13 the killed daemon is /bin/wad: status=0x0 217: 2022-09-12 06:26:14 the killed daemon is /bin/wad: status=0x0 218: 2022-09-12 12:26:15 the killed daemon is /bin/wad: status=0x0 219: 2022-09-12 18:26:15 the killed daemon is /bin/wad: status=0x0 220: 2022-09-13 00:26:15 the killed daemon is /bin/wad: status=0x0 221: 2022-09-13 06:26:16 the killed daemon is /bin/wad: status=0x0 222: 2022-09-13 12:26:16 the killed daemon is /bi

[gfleming]

diagnose sys top  command will show you top CPU usage by process name. Let us know what shows up here and we can further diagnose it.

[Mohammed-Mustafa]

I'm still trying to get results from "diagnose sys top", but think it has something to do with wad process.

[gfleming]

---

@Mohammed-Mustafa wrote:

 but think it has but think it has something to do with wad process.

---

That would make sense given the process is crashing. 

How long does the CPU spike last for?

[Mohammed-Mustafa]

Thanks, Graham, for your responses.

the spike last only for few minutes and then everything works fine, the biggest issue is that there's no high load on the device leading to such behavior. I tired upgrading OS as this problem reported by some users but still no luck.

[Mohammed-Mustafa]

This is the specific process "wad-http(s)-mapi" causing the High CPU.

[gfleming]

Do you have any policies doing proxy-based (not flow-based) inspection?

Do you have any VIPs doing DNAT?

You say there is no "high load" when it happens but what is the traffic load during the CPU spike? Is it different than baseline?

[mr_vaughn]

Check your CPU credits on the AWS monitoring of the EC2 instance. You may have too small of an instance. Usually we go c5.large as a minimum. As per FGT guide for AWS instance sizing.