High CPU in AWS Fortigate t2-small instnace

Mohammed-Mustafa · ‎11-02-2022

have Fortigate OS version 7.2.2 running on AWS using t2.small instance, it periodically has a high CPU, this ccurs at time there's no actual load on the firewall, how to slove this???

I have the following log:

Description:

########## script name: autod.57 ##########
========== #1, 2022-11-02 03:50:07 ==========
FGT $ diagnose debug cli 8
Debug messages will be on for 30 minutes.
FGT $ diagnose debug console timestamp enable
FGT $ diagnose debug enable
FGT $ diagnose debug crashlog read
2: 2022-08-10 17:59:49 (Release)
3: 2022-08-10 17:59:49 <21108> application wad
4: 2022-08-10 17:59:49 <21108> *** signal 11 (Segmentation fault) received ***
5: 2022-08-10 17:59:49 <21108> AVDB 90.04943(08/10/0022 07:26)
6: 2022-08-10 17:59:49 <21108> ETDB 90.04943(08/10/0022 07:26)
7: 2022-08-10 17:59:49 <21108> AVSO 04000000AVEN00701062732202160208
8: 2022-08-10 17:59:49 <21108> Register dump:
9: 2022-08-10 17:59:49 <21108> RAX: fffffffffffffffc RBX: 000000000000000f
10: 2022-08-10 17:59:49 <21108> RCX: ffffffffffffffff RDX: 0000000000000039
11: 2022-08-10 17:59:49 <21108> R08: 000000000000037a R09: 0000000000000009
12: 2022-08-10 17:59:49 <21108> R10: 000000000000037a R11: 0000000000000246
13: 2022-08-10 17:59:49 <21108> R12: 0000000000000001 R13: 0000000000000002
14: 2022-08-10 17:59:49 <21108> R14: 0000000000000002 R15: 0000000000000003
15: 2022-08-10 17:59:49 <21108> RSI: 00000000104bf7d0 RDI: 000000000000000a
16: 2022-08-10 17:59:49 <21108> RBP: 00007fff815dc220 RSP: 00007fff815dc208
17: 2022-08-10 17:59:49 <21108> RIP: 00007f081cba3016 EFLAGS: 0000000000000246
18: 2022-08-10 17:59:49 <21108> CS: 0033 FS: 0000 GS: 0000
19: 2022-08-10 17:59:49 <21108> Trap: 0000000000000000 Error: 0000000000000000
20: 2022-08-10 17:59:49 <21108> OldMask: 0000000000000000
21: 2022-08-10 17:59:49 <21108> CR2: 0000000000000000
22: 2022-08-10 17:59:49 <21108> stack: 0x7fff815dc208 - 0x7fff815dc450
23: 2022-08-10 17:59:49 <21108> Backtrace:
24: 2022-08-10 17:59:49 <21108> [0x7f081cba3016] => /usr/lib/x86_64-linux-gnu/libc.so.6
25: 2022-08-10 17:59:49 (epoll_wait+0x00000016) liboffset 000fa016
26: 2022-08-10 17:59:49 <21108> [0x01b970ea] => /bin/wad
27: 2022-08-10 17:59:49 <21108> [0x01c37416] => /bin/wad
28: 2022-08-10 17:59:49 <21108> [0x0044c767] => /bin/wad
29: 2022-08-10 17:59:49 <21108> [0x7f081caccdeb] => /usr/lib/x86_64-linux-gnu/libc.so.6
30: 2022-08-10 17:59:49 (__libc_start_main+0x000000eb) liboffset 00023deb
31: 2022-08-10 17:59:49 <21108> [0x00447dba] => /bin/wad
32: 2022-08-10 17:59:49 <21108> fortidev 6.0.1.0005
33: 2022-08-10 17:59:49 <21108> process=wad type=2 idx=0 av-scanning=no total=2010 free=324
34: 2022-08-10 17:59:49 mmu=2530767 mu=2502011 m=52957 f=23129 r=0
35: 2022-08-10 17:59:49 <21108> cur_bank=(nil) curl_tl=0xbf731e0 curl_tm=0x7f07fe0ed048
36: 2022-08-10 17:59:49 <21108> (session info)
37: 2022-08-10 17:59:49 [AV Engine <21108>] AV Engine version: 6.4.273
38: 2022-08-10 17:59:49 [AV Engine <21108>] Last file info:
39: 2022-08-10 17:59:49 [AV Engine <21108>] filename: , filesize: 0, filebuffer: (nil)
40: 2022-08-10 17:59:49 [AV Engine <21108>] Native script imagebase: 0x7f07fd411000
41: 2022-08-10 17:59:49 [AV Engine <21108>] Native script imagesize: 0x8000
42: 2022-08-10 17:59:49 [AV Engine <21108>] AV Engine imagebase: 0x7f07fe11e000
43: 2022-08-10 18:59:53 wad crashed 1 times. The latest crash was at 2022-08-10 17:59:49.
44: 2022-08-11 12:25:04 the killed daemon is /bin/wad: status=0x0
45: 2022-08-11 18:25:05 the killed daemon is /bin/wad: status=0x0
46: 2022-08-12 00:25:05 the killed daemon is /bin/wad: status=0x0
47: 2022-08-12 06:25:06 the killed daemon is /bin/wad: status=0x0
48: 2022-08-12 12:25:07 the killed daemon is /bin/wad: status=0x0
49: 2022-08-12 18:25:07 the killed daemon is /bin/wad: status=0x0
50: 2022-08-13 00:25:07 the killed daemon is /bin/wad: status=0x0
51: 2022-08-13 06:25:08 the killed daemon is /bin/wad: status=0x0
52: 2022-08-13 12:25:09 the killed daemon is /bin/wad: status=0x0
53: 2022-08-13 18:25:10 the killed daemon is /bin/wad: status=0x0
54: 2022-08-14 00:25:11 the killed daemon is /bin/wad: status=0x0
55: 2022-08-14 06:25:11 the killed daemon is /bin/wad: status=0x0
56: 2022-08-14 12:25:12 the killed daemon is /bin/wad: status=0x0
57: 2022-08-14 18:25:12 the killed daemon is /bin/wad: status=0x0
58: 2022-08-15 00:25:13 the killed daemon is /bin/wad: status=0x0
59: 2022-08-15 06:25:14 the killed daemon is /bin/wad: status=0x0
60: 2022-08-15 12:25:14 the killed daemon is /bin/wad: status=0x0
61: 2022-08-15 18:25:14 the killed daemon is /bin/wad: status=0x0
62: 2022-08-16 00:25:15 the killed daemon is /bin/wad: status=0x0
63: 2022-08-16 06:25:16 the killed daemon is /bin/wad: status=0x0
64: 2022-08-16 12:25:16 the killed daemon is /bin/wad: status=0x0
65: 2022-08-16 18:25:17 the killed daemon is /bin/wad: status=0x0
66: 2022-08-17 00:25:18 the killed daemon is /bin/wad: status=0x0
67: 2022-08-17 02:51:20 the killed daemon is /bin/csfd: status=0x0
68: 2022-08-17 02:51:21 the killed daemon is /bin/eap_proxy: status=0x0
69: 2022-08-17 06:25:18 the killed daemon is /bin/wad: status=0x0
70: 2022-08-17 12:25:19 the killed daemon is /bin/wad: status=0x0
71: 2022-08-17 18:25:20 the killed daemon is /bin/wad: status=0x0
72: 2022-08-18 00:25:20 the killed daemon is /bin/wad: status=0x0
73: 2022-08-18 06:25:21 the killed daemon is /bin/wad: status=0x0
74: 2022-08-18 12:25:21 the killed daemon is /bin/wad: status=0x0
75: 2022-08-18 18:25:22 the killed daemon is /bin/wad: status=0x0
76: 2022-08-19 00:25:22 the killed daemon is /bin/wad: status=0x0
77: 2022-08-19 06:25:23 the killed daemon is /bin/wad: status=0x0
78: 2022-08-19 12:25:24 the killed daemon is /bin/wad: status=0x0
79: 2022-08-19 18:25:24 the killed daemon is /bin/wad: status=0x0
80: 2022-08-20 00:25:24 the killed daemon is /bin/wad: status=0x0
81: 2022-08-20 06:25:25 the killed daemon is /bin/wad: status=0x0
82: 2022-08-20 12:25:25 the killed daemon is /bin/wad: status=0x0
83: 2022-08-20 18:25:26 the killed daemon is /bin/wad: status=0x0
84: 2022-08-21 00:25:27 the killed daemon is /bin/wad: status=0x0
85: 2022-08-21 06:25:27 the killed daemon is /bin/wad: status=0x0
86: 2022-08-21 12:25:28 the killed daemon is /bin/wad: status=0x0
87: 2022-08-21 18:25:28 the killed daemon is /bin/wad: status=0x0
88: 2022-08-22 00:25:29 the killed daemon is /bin/wad: status=0x0
89: 2022-08-22 06:25:30 the killed daemon is /bin/wad: status=0x0
90: 2022-08-22 12:25:31 the killed daemon is /bin/wad: status=0x0
91: 2022-08-22 18:25:31 the killed daemon is /bin/wad: status=0x0
92: 2022-08-23 00:25:32 the killed daemon is /bin/wad: status=0x0
93: 2022-08-23 06:25:32 the killed daemon is /bin/wad: status=0x0
94: 2022-08-23 12:25:32 the killed daemon is /bin/wad: status=0x0
95: 2022-08-23 18:25:33 the killed daemon is /bin/wad: status=0x0
96: 2022-08-24 00:25:33 the killed daemon is /bin/wad: status=0x0
97: 2022-08-24 06:25:34 the killed daemon is /bin/wad: status=0x0
98: 2022-08-24 12:25:34 the killed daemon is /bin/wad: status=0x0
99: 2022-08-24 18:25:34 the killed daemon is /bin/wad: status=0x0
100: 2022-08-25 00:25:35 the killed daemon is /bin/wad: status=0x0
101: 2022-08-25 06:25:35 the killed daemon is /bin/wad: status=0x0
102: 2022-08-25 12:25:36 the killed daemon is /bin/wad: status=0x0
103: 2022-08-25 18:25:37 the killed daemon is /bin/wad: status=0x0
104: 2022-08-26 00:25:38 the killed daemon is /bin/wad: status=0x0
105: 2022-08-26 06:25:38 the killed daemon is /bin/wad: status=0x0
106: 2022-08-26 12:25:39 the killed daemon is /bin/wad: status=0x0
107: 2022-08-26 18:25:39 the killed daemon is /bin/wad: status=0x0
108: 2022-08-27 00:25:39 the killed daemon is /bin/wad: status=0x0
109: 2022-08-27 06:25:40 the killed daemon is /bin/wad: status=0x0
110: 2022-08-27 12:25:40 the killed daemon is /bin/wad: status=0x0
111: 2022-08-27 18:25:40 the killed daemon is /bin/wad: status=0x0
112: 2022-08-28 00:25:41 the killed daemon is /bin/wad: status=0x0
113: 2022-08-28 06:25:42 the killed daemon is /bin/wad: status=0x0
114: 2022-08-28 12:25:43 the killed daemon is /bin/wad: status=0x0
115: 2022-08-28 18:25:43 the killed daemon is /bin/wad: status=0x0
116: 2022-08-29 00:25:43 the killed daemon is /bin/wad: status=0x0
117: 2022-08-29 06:25:44 the killed daemon is /bin/wad: status=0x0
118: 2022-08-29 12:25:45 the killed daemon is /bin/wad: status=0x0
119: 2022-08-29 18:25:45 the killed daemon is /bin/wad: status=0x0
120: 2022-08-30 00:25:46 the killed daemon is /bin/wad: status=0x0
121: 2022-08-30 06:25:47 the killed daemon is /bin/wad: status=0x0
122: 2022-08-30 12:25:47 the killed daemon is /bin/wad: status=0x0
123: 2022-08-30 18:25:47 the killed daemon is /bin/wad: status=0x0
124: 2022-08-31 00:25:48 the killed daemon is /bin/wad: status=0x0
125: 2022-08-31 06:25:49 the killed daemon is /bin/wad: status=0x0
126: 2022-08-31 12:25:50 the killed daemon is /bin/wad: status=0x0
127: 2022-08-31 18:25:50 the killed daemon is /bin/wad: status=0x0
128: 2022-09-01 00:25:51 the killed daemon is /bin/wad: status=0x0
129: 2022-09-01 06:25:51 the killed daemon is /bin/wad: status=0x0
130: 2022-09-01 12:25:51 the killed daemon is /bin/wad: status=0x0
131: 2022-09-01 18:25:52 the killed daemon is /bin/wad: status=0x0
132: 2022-09-02 00:25:52 the killed daemon is /bin/wad: status=0x0
133: 2022-09-02 04:10:03 service=kernel conserve=on total="2010 MB" used="1776 MB" red="1769 MB"
134: 2022-09-02 04:10:03 green="1648 MB" msg="Kernel enters memory conserve mode"
135: 2022-09-02 04:10:04 MemTotal: 2058868 kB
136: 2022-09-02 04:10:04 MemFree: 103836 kB
137: 2022-09-02 04:10:04 Buffers: 168 kB
138: 2022-09-02 04:10:04 Cached: 1119616 kB
139: 2022-09-02 04:10:04 SwapCached: 0 kB
140: 2022-09-02 04:10:04 Active: 820948 kB
141: 2022-09-02 04:10:04 Inactive: 579992 kB
142: 2022-09-02 04:10:04 Active(anon): 820932 kB
143: 2022-09-02 04:10:04 Inactive(anon): 579572 kB
144: 2022-09-02 04:10:04 Active(file): 16 kB
145: 2022-09-02 04:10:04 Inactive(file): 420 kB
146: 2022-09-02 04:10:04 Unevictable: 195476 kB
147: 2022-09-02 04:10:04 Mlocked: 0 kB
148: 2022-09-02 04:10:04 SwapTotal: 0 kB
149: 2022-09-02 04:10:04 SwapFree: 0 kB
150: 2022-09-02 04:10:04 Dirty: 20 kB
151: 2022-09-02 04:10:04 Writeback: 36 kB
152: 2022-09-02 04:10:04 AnonPages: 476592 kB
153: 2022-09-02 04:10:04 Mapped: 195952 kB
154: 2022-09-02 04:10:04 Shmem: 923876 kB
155: 2022-09-02 04:10:04 Slab: 111732 kB
156: 2022-09-02 04:10:04 SReclaimable: 21064 kB
157: 2022-09-02 04:10:04 SUnreclaim: 90668 kB
158: 2022-09-02 04:10:04 KernelStack: 2080 kB
159: 2022-09-02 04:10:04 PageTables: 31940 kB
160: 2022-09-02 04:10:04 NFS_Unstable: 0 kB
161: 2022-09-02 04:10:04 Bounce: 0 kB
162: 2022-09-02 04:10:04 WritebackTmp: 0 kB
163: 2022-09-02 04:10:04 CommitLimit: 1029432 kB
164: 2022-09-02 04:10:04 Committed_AS: 19133880 kB
165: 2022-09-02 04:10:04 VmallocTotal: 34359738367 kB
166: 2022-09-02 04:10:04 VmallocUsed: 138940 kB
167: 2022-09-02 04:10:04 VmallocChunk: 34359528915 kB
168: 2022-09-02 04:10:04 HugePages_Total: 0
169: 2022-09-02 04:10:04 HugePages_Free: 0
170: 2022-09-02 04:10:04 HugePages_Rsvd: 0
171: 2022-09-02 04:10:04 HugePages_Surp: 0
172: 2022-09-02 04:10:04 Hugepagesize: 2048 kB
173: 2022-09-02 04:10:04 DirectMap4k: 4096 kB
174: 2022-09-02 04:10:04 DirectMap2M: 2093056 kB
175: 2022-09-02 04:10:32 service=kernel conserve=exit total="2010 MB" used="1555 MB" red="1769 MB"
176: 2022-09-02 04:10:32 green="1648 MB" msg="Kernel exits memory conserve mode"
177: 2022-09-02 06:25:52 the killed daemon is /bin/wad: status=0x0
178: 2022-09-02 12:25:53 the killed daemon is /bin/wad: status=0x0
179: 2022-09-02 18:25:54 the killed daemon is /bin/wad: status=0x0
180: 2022-09-03 00:25:54 the killed daemon is /bin/wad: status=0x0
181: 2022-09-03 06:25:55 the killed daemon is /bin/wad: status=0x0
182: 2022-09-03 12:25:55 the killed daemon is /bin/wad: status=0x0
183: 2022-09-03 18:25:55 the killed daemon is /bin/wad: status=0x0
184: 2022-09-04 00:25:56 the killed daemon is /bin/wad: status=0x0
185: 2022-09-04 06:25:57 the killed daemon is /bin/wad: status=0x0
186: 2022-09-04 12:25:58 the killed daemon is /bin/wad: status=0x0
187: 2022-09-04 18:25:58 the killed daemon is /bin/wad: status=0x0
188: 2022-09-05 00:25:59 the killed daemon is /bin/wad: status=0x0
189: 2022-09-05 06:26:00 the killed daemon is /bin/wad: status=0x0
190: 2022-09-05 12:26:00 the killed daemon is /bin/wad: status=0x0
191: 2022-09-05 18:26:01 the killed daemon is /bin/wad: status=0x0
192: 2022-09-06 00:26:01 the killed daemon is /bin/wad: status=0x0
193: 2022-09-06 06:26:02 the killed daemon is /bin/wad: status=0x0
194: 2022-09-06 12:26:02 the killed daemon is /bin/wad: status=0x0
195: 2022-09-06 18:26:03 the killed daemon is /bin/wad: status=0x0
196: 2022-09-07 00:26:03 the killed daemon is /bin/wad: status=0x0
197: 2022-09-07 06:26:03 the killed daemon is /bin/wad: status=0x0
198: 2022-09-07 12:26:03 the killed daemon is /bin/wad: status=0x0
199: 2022-09-07 18:26:04 the killed daemon is /bin/wad: status=0x0
200: 2022-09-08 00:26:05 the killed daemon is /bin/wad: status=0x0
201: 2022-09-08 06:26:05 the killed daemon is /bin/wad: status=0x0
202: 2022-09-08 12:26:06 the killed daemon is /bin/wad: status=0x0
203: 2022-09-08 18:26:06 the killed daemon is /bin/wad: status=0x0
204: 2022-09-09 00:26:07 the killed daemon is /bin/wad: status=0x0
205: 2022-09-09 06:26:07 the killed daemon is /bin/wad: status=0x0
206: 2022-09-09 12:26:07 the killed daemon is /bin/wad: status=0x0
207: 2022-09-09 18:26:08 the killed daemon is /bin/wad: status=0x0
208: 2022-09-10 00:26:08 the killed daemon is /bin/wad: status=0x0
209: 2022-09-10 06:26:09 the killed daemon is /bin/wad: status=0x0
210: 2022-09-10 12:26:10 the killed daemon is /bin/wad: status=0x0
211: 2022-09-10 18:26:11 the killed daemon is /bin/wad: status=0x0
212: 2022-09-11 00:26:11 the killed daemon is /bin/wad: status=0x0
213: 2022-09-11 06:26:11 the killed daemon is /bin/wad: status=0x0
214: 2022-09-11 12:26:12 the killed daemon is /bin/wad: status=0x0
215: 2022-09-11 18:26:12 the killed daemon is /bin/wad: status=0x0
216: 2022-09-12 00:26:13 the killed daemon is /bin/wad: status=0x0
217: 2022-09-12 06:26:14 the killed daemon is /bin/wad: status=0x0
218: 2022-09-12 12:26:15 the killed daemon is /bin/wad: status=0x0
219: 2022-09-12 18:26:15 the killed daemon is /bin/wad: status=0x0
220: 2022-09-13 00:26:15 the killed daemon is /bin/wad: status=0x0
221: 2022-09-13 06:26:16 the killed daemon is /bin/wad: status=0x0
222: 2022-09-13 12:26:16 the killed daemon is /bi

gfleming · ‎11-02-2022

diagnose sys top command will show you top CPU usage by process name. Let us know what shows up here and we can further diagnose it.

Cheers,
Graham

Mohammed-Mustafa · ‎11-06-2022

I'm still trying to get results from "diagnose sys top", but think it has something to do with wad process.

gfleming · ‎11-06-2022

@Mohammed-Mustafa wrote:

but think it has but think it has something to do with wad process.

That would make sense given the process is crashing.

How long does the CPU spike last for?

Cheers,
Graham

Mohammed-Mustafa · ‎11-06-2022

Thanks, Graham, for your responses.

the spike last only for few minutes and then everything works fine, the biggest issue is that there's no high load on the device leading to such behavior. I tired upgrading OS as this problem reported by some users but still no luck.

Mohammed-Mustafa · ‎11-15-2022

This is the specific process "wad-http(s)-mapi" causing the High CPU.

gfleming · ‎11-15-2022

Do you have any policies doing proxy-based (not flow-based) inspection?

Do you have any VIPs doing DNAT?

You say there is no "high load" when it happens but what is the traffic load during the CPU spike? Is it different than baseline?

Cheers,
Graham

mr_vaughn · ‎11-16-2022

Check your CPU credits on the AWS monitoring of the EC2 instance. You may have too small of an instance. Usually we go c5.large as a minimum. As per FGT guide for AWS instance sizing.

High CPU in AWS Fortigate t2-small instnace

Nominate a Forum Post for Knowledge Article Creation

You are leaving our website