Creating a Parser

[KarlH]

Hi folks this is my first post here and my first parser in FortiSIEM.  I have the raw logs.

How do I develop the XML in the body, to grab the fields in the log?  Not sure of the approach here.

Hmm Apparently I've managed to post incorrectly. 

I have poured over the docs and I am not getting it.

I have my regexe here:

Account Id":"\[\\"([0-9a-fA-F\-]+)\\
Attack Module":"\[\\"(.*?)\\"
Attack Time":"\[\\"(.*?)\\
Code Processed":"\[\\"(.*?)\\
Command Line":"\[\\"(.*?)\\
Computer Name":"\[\\"([0-9A-F\-]+)\\ OR THIS Computer Name":"\[\\"([^\"]+?)\\"\\]
Logged In UserName":"\[\\"[A-Z]+\/([^\\]+)\\\"\]\" OR POSSIBLY THIS Logged In UserName":"\[\\"[A-Z]+\/(.*?)\\
Message":"\[\\"(.*?)\\"
Morphisec Version":"\[\\"(.*?)\\"
Parent Process Command Line":"\[\\"(.*?)\\"
Protector IP":"\[\\"(.*?)\\"
Threat Description":"\[\\"(.*?)\\"
Threat Module":"\[\\"(.*?)\\"
Threat Name":"\[\\"(.*?)\\"

I have this raw log:

2024-09-26 15:55:15 Morphisec-EPTP INFO MORPHISEC_ATTACK {"Account Id":"[\"0cebd16e-eba3-40a1-a2b6-88e9e20787d3\"]","Attack Module":"[\"kernel32.dll\"]","Attack Time":"[\"2024-09-26T15:54:54.494Z\"]","Code Processed":"[\"0x007ffe63158600 MOV RAX, RSP\"]","Command Line":"[\"C:/Users/shaned/Desktop/Fake Malicious File/winwword.exe\"]","Computer Name":"[\"GWLT011-02075\"]","File Hash":"[\"\"]","File Name":"[\"\"]","Last Module Loaded":"[\"0x00007FFE50830000 | 0x00007FFE50862000 | 0x32000 | 0x20 | C:/WINDOWS/SYSTEM32/dbgcore.DLL (FileDescription:Windows Core Debugging Helpers;ProductName:Microsoft® Windows® Operating System;VersionInfo:10.0.22621.1 (WinBuild.160101.0800);Timestamp:Sun Mar 9 22:12:15 1980;ASLR:Enabled)\"]","Last Stack FunctionCall":"[\"kernel32.dll| 0x0000000000068600 ( WinExec) | 0x00007FFE630F0000\"]","Logged In UserName":"[\"GWNT/shane.dickson\"]","Message":"[\"Morphisec prevented a threat on application winwword\"]","Morphisec Version":"[\"8.3.2\"]","Parent Process Command Line":"[\"C:/Windows/explorer.exe\"]","Parent Signature":"[\"359179ffb630953ee79523866a0a2246a5612d726c2eace52f7413f15530715e\"]","Process Signature":"[\"d3d97b6af2457c9a8c43cb3856ff227802dc928836638d8e71258c9167379168\"]","Protector IP":"[\"192.168.0.139\"]","Tenant Id":"[\"69068d46-d11d-4e81-af93-0b91bec183ef\"]","Threat Description":"[\"Morphisec Total Evasion Framework is a tool that contains several Pen-testing attack techniques that bypass all AV and EDR solutions.\"]","Threat Module":"[\"Shellcode\"]","Threat Name":"[\"Morphisec Total Evasion Framework\"]","Threat Severity":"[\"%!s(int=5)\"]","Threat Sub-Classification":"[\"Attack-Simulator\

I also have all the regular expressions to pull the fields.

1) I do not understand how to populate the Details pane in analytics with new labels and field values

2) I also do not understand the laws governing the xml.  Can anyone tell me based on the raw log I provided, ifI am to use

3) I'm just not getting how I embed the regex into this xml,  do I use a function? or do I use this statement

<collectFieldsByRegex src="$_rawmsg">    and how does that $ variable get populated, with the raw log I assume?

Take this example below, what does patApachMod mean? what I use based on my log where did all the pvariable come 
from?  patQuote and patEXcpetion why did they call it that? how do I map my values to readable fields the analysts 
can read in the gui.
 Any advice would be welcome.  its just not clear. 
 This is excruciating. 
<patternDefinitions>
<pattern name="patApacheMod"><![CDATA[Apache_AccessLog:|Apache_ErrorLog:|ApacheLog:?|HTTP:|httpd:]]></pattern>
<pattern name="patQuote"><![CDATA[[^"]*]]></pattern>
<pattern name="patExceptColon"><![CDATA[[^:]+]]></pattern>
<pattern name="patExceptComma"><![CDATA[[^,]+]]></pattern>
</patternDefinitions>
<

<eventFormatRecognizer><![CDATA[<:gPatSyslogPRI>\s*<:gPatMon>\s+<:gPatDay>\s+<:gPatTime>\s+<:gPatStr>\s+<:patApacheMod>\s+(?:\d+\s+)?(?:<:gPatIpAddr>|\[<:gPatStr>)]]></eventFormatRecognizer>

then I see the sections are in different orders from parser to parser.

Finally how is my Mophisec rawlog getting parsed now even thought its says unknownlog source its still getting parse
somehow and populating the fields in Detail pane. 

If someone could please answer some of these questions I would appreciate it. 

Thank you, Karl

[KarlH · Accepted solution]

I have all my regular expressions via regex101.com that correctly extract the fields but now how to add them to the xml body.

So here is the raw log :

2024-09-26 15:55:15 Morphisec-EPTP INFO MORPHISEC_ATTACK {"Account Id":"[\"0cebd16e-eba3-40a1-a2b6-88e9e20787d3\"]","Attack Module":"[\"kernel32.dll\"]","Attack Time":"[\"2024-09-26T15:54:54.494Z\"]","Code Processed":"[\"0x007ffe63158600 MOV RAX, RSP\"]","Command Line":"[\"C:/Users/shaned/Desktop/Fake Malicious File/winwword.exe\"]","Computer Name":"[\"GWLT011-02075\"]","File Hash":"[\"\"]","File Name":"[\"\"]","Last Module Loaded":"[\"0x00007FFE50830000 | 0x00007FFE50862000 | 0x32000 | 0x20 | C:/WINDOWS/SYSTEM32/dbgcore.DLL (FileDescription:Windows Core Debugging Helpers;ProductName:Microsoft® Windows® Operating System;VersionInfo:10.0.22621.1 (WinBuild.160101.0800);Timestamp:Sun Mar 9 22:12:15 1980;ASLR:Enabled)\"]","Last Stack FunctionCall":"[\"kernel32.dll| 0x0000000000068600 ( WinExec) | 0x00007FFE630F0000\"]","Logged In UserName":"[\"GWNT/shane.dickson\"]","Message":"[\"Morphisec prevented a threat on application winwword\"]","Morphisec Version":"[\"8.3.2\"]","Parent Process Command Line":"[\"C:/Windows/explorer.exe\"]","Parent Signature":"[\"359179ffb630953ee79523866a0a2246a5612d726c2eace52f7413f15530715e\"]","Process Signature":"[\"d3d97b6af2457c9a8c43cb3856ff227802dc928836638d8e71258c9167379168\"]","Protector IP":"[\"192.168.0.139\"]","Tenant Id":"[\"69068d46-d11d-4e81-af93-0b91bec183ef\"]","Threat Description":"[\"Morphisec Total Evasion Framework is a tool that contains several Pen-testing attack techniques that bypass all AV and EDR solutions.\"]","Threat Module":"[\"Shellcode\"]","Threat Name":"[\"Morphisec Total Evasion Framework\"]","Threat Severity":"[\"%!s(int=5)\"]","Threat Sub-Classification":"[\"Attack-Simulator\

I have regular expressions for every field in it. Its crafting the xml that I'm not clear on.

[cdurkin_FTNT · Accepted solution]

Try this ... for starters ..  its a partial shell for the one log message (which was not a full log)..

Look up the FortiSIEM Parser Training on https://training.fortinet.com/ for some reference material.

[KarlH · Accepted solution]

I have all my regular expressions via regex101.com that correctly extract the fields but now how to add them to the xml body.

So here is the raw log :

2024-09-26 15:55:15 Morphisec-EPTP INFO MORPHISEC_ATTACK {"Account Id":"[\"0cebd16e-eba3-40a1-a2b6-88e9e20787d3\"]","Attack Module":"[\"kernel32.dll\"]","Attack Time":"[\"2024-09-26T15:54:54.494Z\"]","Code Processed":"[\"0x007ffe63158600 MOV RAX, RSP\"]","Command Line":"[\"C:/Users/shaned/Desktop/Fake Malicious File/winwword.exe\"]","Computer Name":"[\"GWLT011-02075\"]","File Hash":"[\"\"]","File Name":"[\"\"]","Last Module Loaded":"[\"0x00007FFE50830000 | 0x00007FFE50862000 | 0x32000 | 0x20 | C:/WINDOWS/SYSTEM32/dbgcore.DLL (FileDescription:Windows Core Debugging Helpers;ProductName:Microsoft® Windows® Operating System;VersionInfo:10.0.22621.1 (WinBuild.160101.0800);Timestamp:Sun Mar 9 22:12:15 1980;ASLR:Enabled)\"]","Last Stack FunctionCall":"[\"kernel32.dll| 0x0000000000068600 ( WinExec) | 0x00007FFE630F0000\"]","Logged In UserName":"[\"GWNT/shane.dickson\"]","Message":"[\"Morphisec prevented a threat on application winwword\"]","Morphisec Version":"[\"8.3.2\"]","Parent Process Command Line":"[\"C:/Windows/explorer.exe\"]","Parent Signature":"[\"359179ffb630953ee79523866a0a2246a5612d726c2eace52f7413f15530715e\"]","Process Signature":"[\"d3d97b6af2457c9a8c43cb3856ff227802dc928836638d8e71258c9167379168\"]","Protector IP":"[\"192.168.0.139\"]","Tenant Id":"[\"69068d46-d11d-4e81-af93-0b91bec183ef\"]","Threat Description":"[\"Morphisec Total Evasion Framework is a tool that contains several Pen-testing attack techniques that bypass all AV and EDR solutions.\"]","Threat Module":"[\"Shellcode\"]","Threat Name":"[\"Morphisec Total Evasion Framework\"]","Threat Severity":"[\"%!s(int=5)\"]","Threat Sub-Classification":"[\"Attack-Simulator\

I have regular expressions for every field in it. Its crafting the xml that I'm not clear on.

[cdurkin_FTNT · Accepted solution]

Try this ... for starters ..  its a partial shell for the one log message (which was not a full log)..

Look up the FortiSIEM Parser Training on https://training.fortinet.com/ for some reference material.

[KarlH]

Thank you @Rob_SIEM  you it now validate and tests successfully. Man I could not have done this with out you, I think Morphisec is working on that Threat Severity":"[\"%!s(int=5)\]    its not supposed to look like that.  Thank you sir!   robit now validate and tests successfully. Man I could not have done this with out you, I think Morphisec is working on that Threat Severity":"[\"%!s(int=5)\]    its not supposed to look like that.  Thank you sir!   

[KarlH · Accepted solution]

So what is missing from the docs is the simple steps of taking the fields from the raw log  and developing the regular expressions then simply mapping them to the attributes.  That's where you begin to present the data on the Details pane. From here you go to the Events Attributes tab under Admin, --> Device support and then Events Attributes.  The goal of parsing is data representation and presentation into a readable form for the audience? the details screen. All the docs about xml and the rest forgot how to explain the steps.Sorry to say. 

The steps of creating as parser start with extracting the fields and identifying the canned field and somnetime the custom ones in the Event Attributes  tab.

The values on the left are the real field names

from the regex

Account Id":"\[\\"([0-9a-fA-F\-]+)\\ ----- for example

The value on the right side of the equals is the 'Name' column' over in the Event Attributes. thats were the details pane gets its fields. from the values on the left.

Account Id - accountid
Attack Module - morphisecAttackModule
Attack time - accessTime
Code processed - morphisecCodeProcessed
Command Line - morphisecCommandLine
Computer Name - targetComputer
Logged In UserNam - log ID
Message - actionResult
Morphisec Version - morphisecVersion
Parent Process Command Line - ProcessName
Protector IP - morphisecProtectorIP
Threat Description - serviceDesc
Threat Module - module
Threat Name - threatInfoReliability

I think I will keep this open to share with others who are struggling with the process of custom parsing in FortiSiem.