New Member

Solved

FortiOS CLI question

Forum|Forum|11 years ago
April 20, 2015
13 replies
57558 views

We are running an old version of FortiOS 4.3 (patch 6) with a known memory leak. Obviously it needs to be updated. In the meantime, once a month one of the network engineers was killing the rogue process to free up the memory. He has since left the company and didn't document what the process was or how to kill it. All I have is a Fortinet ticket #.

I connected to the CLI but the only CLI commands available (both via web and ssh) are config, get, show and exit. I'm looking at the FortiOS Handbook CLI Reference for FortiOS 4.3 and is says the command I should use is "system performance top". However "system" isn't valid (5499: Unknown action 0 Command fail. Return code -1)

What might be the reason "system" isn't available?

Best answer by Christopher_McMullan

The general syntax for the CLI is verb-area-noun, so every command has to start with config, execute, get, show, or diagnose.

In this case, the command to view 'top' data as in Linux would be 'diag sys top'. You can also get a system performance snapshot with 'get sys perf stat'.

If 4.3.6 is suffering from merged_daemons, you would want to run 'diag sys top', and immediately press 'q' afterwards to generate one set of results. Find the process ID for merged_daemons (if that's truly the offending process - but from that build, it likely is), then run 'diag sys kill 11 <PID>'. For instance, if merged_daemons is running with a PID of 50, the command would be 'diag sys kill 11 50'.

The CLI reference guide, except for the bottom sections dealing with the commands beginning with the verbs 'get' and 'execute' all assume an initial verb of 'config'. That may be where the confusion was introduced: every section like 'alertemail...' or 'router....' assumes it begins with 'config'. So, for static routes, the document path would be router > static, but the full command would be 'config router static'.

Christopher_McMullanAnswer

Staff

The general syntax for the CLI is verb-area-noun, so every command has to start with config, execute, get, show, or diagnose.

In this case, the command to view 'top' data as in Linux would be 'diag sys top'. You can also get a system performance snapshot with 'get sys perf stat'.

If 4.3.6 is suffering from merged_daemons, you would want to run 'diag sys top', and immediately press 'q' afterwards to generate one set of results. Find the process ID for merged_daemons (if that's truly the offending process - but from that build, it likely is), then run 'diag sys kill 11 <PID>'. For instance, if merged_daemons is running with a PID of 50, the command would be 'diag sys kill 11 50'.

The CLI reference guide, except for the bottom sections dealing with the commands beginning with the verbs 'get' and 'execute' all assume an initial verb of 'config'. That may be where the confusion was introduced: every section like 'alertemail...' or 'router....' assumes it begins with 'config'. So, for static routes, the document path would be router > static, but the full command would be 'config router static'.

rawbarAuthor

New Member

Thanks very much for the quick and thorough explanation. However diag is not a valid command for me nor is system. Both generate 5499: Unknown action.

If I hit ? to get a list of valid command, the only ones listed are config, get, show and exit.

'get sys perf stat' also is not valid. I can do a 'get system status' but for get system, the only valid options I'm shown with 'get system ?' are 'status' and 'system status'.

Could it be a permission on this account issue? I'm using what should be a root account, but it's entirely possible someone in our EU team has limited the permission on the US root account. Update: I just checked and this account is assigned to the 'super_admin' profile, same as the root account.

rawbarAuthor

New Member

The profile is super_admin

rawbarAuthor

New Member

That doesn't seem to be the issue unless something is wrong with the super_admin profile. My account is assigned to super_admin, and I just checked super_admin permissions and everything is read/write across the board.

Christopher_McMullan

Staff

It seems like a permissions issue. You might be able to see what profile has been applied to your account:

config system admin

edit <admin_name>

show

end

If the accprofile is prof_admin, or anything other than super_admin, restrictions are likely being applied.

Christopher_McMullan

Staff

One solution would be to use the maintainer account to recover the super admin's password, if you have the scope to:

config system global

get | grep admin-maintainer

end

If admin-maintainer is enabled, this is equivalent to changing the boot variables for Cisco devices from 0x2102 (from memory, this is normal). Use a console connection, and immediately after gaining the login prompt, you have a short amount of time to login as:

user maintainer

password bcpbserialnumberoffortigate

For instance, my old 80C had the serial number FGT80Cxxxxxx5328. I would enter:

user maintainer

pass bcpbFGT80Cxxxxxx5328 (case sensitive)

This would grant me super user access to the CLI, where I could view and modify the admin accounts, admin profiles, passwords, etc. in order to regain root-level permissions.

rawbarAuthor

New Member

This account is super_admin

2015-04-20_1029.jpg

Christopher_McMullan

Staff

I've only seen references to that specific error when an HA cluster was involved.

It may be worth your while to boot into maintainer anyway, to see if you still are locked out of 'diagnose' commands. If 'diag...' is available with maintainer, you could try creating a new admin account to sidestep the issues with the existing admin users. If 'diagnose' is still unavailable, it may point to deeper corruption.

rawbarAuthor

New Member

This is indeed an HA cluster. I'm ssh'd into the master.

Christopher_McMullan

Staff

Okay...I think the best way forward is to open a ticket with Fortinet TAC to explore cluster issues. Explain your access limitations, and see what can be done short of immediately upgrading.

The solutions in the related tickets for HA clusters encountering the output you posted aren't consistent, so I can't provide a catch-all fix you could try without further analysis.

rawbarAuthor

New Member

OK! For anybody else that runs into this.... I needed to first type:

config vdom

edit root

NOW diagnose is available to me.

Christopher_McMullan

Staff

VDOMs...<foot inserted into mouth>

I can over-think things - I haven't seen that error come up when VDOMs are present and we don't enter the context of a VDOM first. That may explain why more tickets don't note the error as an issue.

Glad it's not something more serious.

rawbarAuthor

New Member

Thanks again for the help. In this case, the offending process was forticron. Once I killed the process, I was then able to save config changes I'd been trying to make. However the documented method of killing the offending process did not work for me.

diag sys kill 11 <PID> gave me: "command parse error before 'kill'

I ended up killing the process with: fnsysctl kill -9 PID

Christopher McMullan_FTNT wrote:
VDOMs...<foot inserted into mouth>

I can over-think things - I haven't seen that error come up when VDOMs are present and we don't enter the context of a VDOM first. That may explain why more tickets don't note the error as an issue.

Glad it's not something more serious.

Sign up

Login with SSO

Login to the community

Login with SSO