[FortiSIEM]How to Receive JSON Logs from Other Systems passively

[Bruce7x2]

Dear Team,

Based on the article "Ingesting JSON Formatted Events Received via HTTP(S) POST", I attempted to import logs from my DHCP Server (note: these are actual log files, not real-time logs sent by the DHCP Server). However, I encountered several issues, which are described below:

1.I'm currently using curl to send a JSON file (dhcpactivity.json) to FortiSIEM via HTTP (not HTTPS), including the -u '<admin>:<password>' information. However, the result returns a 401 Unauthorized error. What should I do about this? Or do I need to create a dedicated account in FortiSIEM (which I've actually done, but it still doesn't work)?
2.If I'm using HTTP, do I need to add the line "htpasswd -b /etc/httpd/accounts/passwds <user> '<password>'"? Or under what circumstances should this be added? (I've actually added this as well)
3.Before executing the curl command to send the logs, do I need to define Parsers in FortiSIEM first?
4.I noticed that the log example in the article (Ingesting JSON Formatted Events Received via HTTP(S) POST) is [PH_DEV_MON_CUSTOM_JSON]:[reptVendor]=<vendor>,[reptModel]=<model>,[reptDevName]=<reptName>,[reptDevIpAddr]=<reptIp>,[json]=<JSON>.
However, my log looks like the content shown below:

{"data":{"dhcpv4Message":{"chaddr":"00:00:00:00:00:01","siaddr":"0.0.0.0","secs":0,"hType":1,"xId":"0xc123456d","yiaddr":"192.168.1.1","giaddr":"192.168.1.2","options":[{"optionId":53,"messageType":"DHCP Ack","messageTypeId":5},{"serverId":"192.168.1.10","optionId":54},{"optionId":51,"leaseTime":67556},{"subnetMask":"255.255.255.0","optionId":1},{"router":["192.168.1.100"],"optionId":3},{"domainNameSever":["192.168.1.10","192.168.1.20","192.168.1.30"],"optionId":6},{"domainName":"domain.local","optionId":15},{"netBIOSNameServer":["192.168.1.40"],"optionId":44},{"optionId":46,"netBIOSNodeType":"H-node"},{"optionData":"00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00","optionId":81},{"optionData":"00 00 00 00 00 00 00 00 00 00 00 00 00 00 00","optionId":82}],"flags":"0x0000","hops":1,"ciaddr":"0.0.0.0","hLength":6,"op":2}},"schemaVersion":"1.0.0","headers":{"http_version":"HTTP/1.1","content_length":"37932","content_type":"application/json","request_method":"POST","http_user_agent":"Vector/0.22.1 (x86_64-unknown-linux-gnu)","http_host":"192.168.1.101:80","http_accept":null,"request_path":"/","accept_encoding":"identity"},"time":1719422428,"sourceId":"00000000-0000-0000-0000-000000000000","type":"dhcpactivity","timePrecision":"s","key":"00000000-0000-0000-0000-000000000000","@version":"1","tags":["_geoip_lookup_failure"],"payloadType":"dhcpv4-packet","host":"192.168.1.17","serverId":"dhcp-01","@timestamp":"2024-06-26T17:20:11.570Z"}
Can FortiSIEM parse this log content without modifying the Parser?

References:
[1] https://docs.fortinet.com/document/fortisiem/7.2.1/external-systems-configuration-guide/522142/inges...
[2] https://help.fortinet.com/fsiem/6-4-0/Online-Help/HTML5_Help/Ingesting_JSON.htm

[Bruce7x2]

Dear Rob,

I greatly appreciate your professional assistance, which has provided me with very useful information. I am making progress towards my goal step by step (normalizing this DHCP JSON log and displaying relevant information through the Dashboard).
Currently, I can send DHCP logs to FortiSIEM via HTTPS with the addition of a Bearer Token.Of course, I have also created Devices/Apps according to your instructions.Now, I can obtain the format of 

[PH_DEV_MON_CUSTOM_JSON]:[reptVendor]=BlueCat,[reptModel]=AddressManager,[reptDevName]=bluecat-dhcp01,[reptDevIpAddr]=1.1.1.1,[json]={“data”:{“dhcpv4Message”:…

through the Raw Event Log.

I believe this is key to creating a Parser, right?

Assuming my log format is as follows:

{“data”:{“dhcpv4Message”:{“chaddr”:“00:00:00:00:00:01”,“siaddr”:“0.0.0.0”,“secs”:0,“hType”:1,“xId”:“0xc123456d”,“yiaddr”:“192.168.1.1”,“giaddr”:“192.168.1.2”,“options”:[{“optionId”:53,“messageType”:“DHCP Ack”,“messageTypeId”:5},{“serverId”:“192.168.1.10”,“optionId”:54},{“optionId”:51,“leaseTime”:67556},{“subnetMask”:“255.255.255.0”,“optionId”:1},{“router”:[“192.168.1.100”],“optionId”:3},{“domainNameSever”:[“192.168.1.10”,“192.168.1.20”,“192.168.1.30”],“optionId”:6},{“domainName”:“domain.local”,“optionId”:15},{“netBIOSNameServer”:[“192.168.1.40”],“optionId”:44},{“optionId”:46,“netBIOSNodeType”:“H-node”},{“optionData”:“00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00”,“optionId”:81},{“optionData”:“00 00 00 00 00 00 00 00 00 00 00 00 00 00 00”,“optionId”:82}],“flags”:“0x0000”,“hops”:1,“ciaddr”:“0.0.0.0”,“hLength”:6,“op”:2}},“schemaVersion”:“1.0.0”,“headers”:{“http_version”:“HTTP/1.1”,“content_length”:“37932”,“content_type”:“application/json”,“request_method”:“POST”,“http_user_agent”:“Vector/0.22.1 (x86_64-unknown-linux-gnu)”,“http_host”:“192.168.1.101:80”,“http_accept”:null,“request_path”:“/”,“accept_encoding”:“identity”},“time”:1719422428,“sourceId”:“00000000-0000-0000-0000-000000000000”,“type”:“dhcpactivity”,“timePrecision”:“s”,“key”:“00000000-0000-0000-0000-000000000000”,“@version”:“1”,“tags”:[“_geoip_lookup_failure”],“payloadType”:“dhcpv4-packet”,“host”:“192.168.1.17”,“serverId”:“dhcp-01”,“@timestamp”:“2024-06-26T17:20:11.570Z”}

Now, I want to fetch (normalize) the ‘yiaddr’ field and the ‘serverId’ field from the log. But when I tried to parse the PHCustomJsonParser file and made some attempts, I got incorrect results.

Therefore, I hope you can provide me with some examples, or give me some suggestions for modifying my content, so that I can understand and add normalized content on my own.

I am currently trying to modify the content of PHCustomJsonParser as follows, but it failed when executing the Test.

<!-- attribute mapping -->

<collectAndSetAttrByJSON src="$_json">

<attrKeyMap attr="data.dhcpv4Message.chaddr" key="chaddr"/>

<attrKeyMap attr="data.dhcpv4Message.siaddr" key="siaddr"/>

</collectAndSetAttrByJSON>

<!-- extra parsing -->

<when test="$reptVendor = 'BlueCat'">

<when test="$reptModel = 'AddressMessager'">

<when test="exist _logId">

<setEventAttribute attr="eventType">combineMsgId("BlueCat-JSON-", $_logId)</setEventAttribute>

</when>

</when>

</when>

Please give me some suggestions for modifying PHCustomJsonParser:

[cdurkin_FTNT]

You can build you own parser for this..

I threw your test event into a parserTool I use, and this should get you started.

You will need to define the attributes you wish to extract.

It wasn't clear what field you wanted to use a _LogId, so I chose one at random, and Source IP is extracted as an example ..

<eventFormatRecognizer><![CDATA[\[PH_DEV_MON_CUSTOM_JSON\]:\[reptVendor\]=BlueCat,\[reptModel\]=AddressManager,]]></eventFormatRecognizer>

<parsingInstructions>

<collectFieldsByRegex src="$_rawmsg">
  <regex><![CDATA[\[PH_DEV_MON_CUSTOM_JSON\]:\[reptVendor\]=BlueCat,\[reptModel\]=AddressManager,\[reptDevName\]=<reptDevName:gPatStrComma>,\[reptDevIpAddr\]=<_reptDevIp:gPatIpV4Dot>,\[json\]=<_json:gPatMesgBody>]]></regex>
</collectFieldsByRegex>
<setEventAttribute attr="eventType">BlueCat-JSON-Generic</setEventAttribute>


<!-- Test Event
[PH_DEV_MON_CUSTOM_JSON]:[reptVendor]=BlueCat,[reptModel]=AddressManager,[reptDevName]=bluecat-dhcp01,[reptDevIpAddr]=1.1.1.1,[json]={"data":{"dhcpv4Message":{"chaddr":"00:00:00:00:00:01","siaddr":"0.0.0.0","secs":0,"hType":1,"xId":"0xc123456d","yiaddr":"192.168.1.1","giaddr":"192.168.1.2","options":[{"optionId":53,"messageType":"DHCP Ack","messageTypeId":5},{"serverId":"192.168.1.10","optionId":54},{"optionId":51,"leaseTime":67556},{"subnetMask":"255.255.255.0","optionId":1},{"router":["192.168.1.100"],"optionId":3},{"domainNameSever":["192.168.1.10","192.168.1.20","192.168.1.30"],"optionId":6},{"domainName":"domain.local","optionId":15},{"netBIOSNameServer":["192.168.1.40"],"optionId":44},{"optionId":46,"netBIOSNodeType":"H-node"},{"optionData":"00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00","optionId":81},{"optionData":"00 00 00 00 00 00 00 00 00 00 00 00 00 00 00","optionId":82}],"flags":"0x0000","hops":1,"ciaddr":"0.0.0.0","hLength":6,"op":2}},"schemaVersion":"1.0.0","headers":{"http_version":"HTTP/1.1","content_length":"37932","content_type":"application/json","request_method":"POST","http_user_agent":"Vector/0.22.1 x86_64-unknown-linux-gnu","http_host":"192.168.1.101:80","http_accept":"xxx","request_path":"/","accept_encoding":"identity"},"time":1719422428,"sourceId":"00000000-0000-0000-0000-000000000000","type":"dhcpactivity","timePrecision":"s","key":"00000000-0000-0000-0000-000000000000","@version":"1","tags":["_geoip_lookup_failure"],"payloadType":"dhcpv4-packet","host":"192.168.1.17","serverId":"dhcp-01","@timestamp":"2024-06-26T17:20:11.570Z"}
-->

<collectAndSetAttrByJSON src="$_json">
	<attrKeyMap attr="_chaddr" key="data.dhcpv4Message.chaddr"/>
	<attrKeyMap attr="_siaddr" key="data.dhcpv4Message.siaddr"/>
	<attrKeyMap attr="_secs" key="data.dhcpv4Message.secs"/>
	<attrKeyMap attr="_logId" key="data.dhcpv4Message.hType"/>  <!-- random -->
	<attrKeyMap attr="_xId" key="data.dhcpv4Message.xId"/>
	<attrKeyMap attr="srcIpAddr" key="data.dhcpv4Message.yiaddr"/> <!-- random -->
	<attrKeyMap attr="_giaddr" key="data.dhcpv4Message.giaddr"/>
	<attrKeyMap attr="_optionId" key="data.dhcpv4Message.options[0].optionId"/>
	<attrKeyMap attr="_messageType" key="data.dhcpv4Message.options[0].messageType"/>
	<attrKeyMap attr="_messageTypeId" key="data.dhcpv4Message.options[0].messageTypeId"/>
	<attrKeyMap attr="_serverId" key="data.dhcpv4Message.options[1].serverId"/>
	<attrKeyMap attr="_optionId" key="data.dhcpv4Message.options[1].optionId"/>
	<attrKeyMap attr="_optionId" key="data.dhcpv4Message.options[2].optionId"/>
	<attrKeyMap attr="_leaseTime" key="data.dhcpv4Message.options[2].leaseTime"/>
	<attrKeyMap attr="_subnetMask" key="data.dhcpv4Message.options[3].subnetMask"/>
	<attrKeyMap attr="_optionId" key="data.dhcpv4Message.options[3].optionId"/>
	<attrKeyMap attr="_router[0]" key="data.dhcpv4Message.options[4].router[0]"/>
	<attrKeyMap attr="_optionId" key="data.dhcpv4Message.options[4].optionId"/>
	<attrKeyMap attr="_domainNameSever[0]" key="data.dhcpv4Message.options[5].domainNameSever[0]"/>
	<attrKeyMap attr="_domainNameSever[1]" key="data.dhcpv4Message.options[5].domainNameSever[1]"/>
	<attrKeyMap attr="_domainNameSever[2]" key="data.dhcpv4Message.options[5].domainNameSever[2]"/>
	<attrKeyMap attr="_optionId" key="data.dhcpv4Message.options[5].optionId"/>
	<attrKeyMap attr="_domainName" key="data.dhcpv4Message.options[6].domainName"/>
	<attrKeyMap attr="_optionId" key="data.dhcpv4Message.options[6].optionId"/>
	<attrKeyMap attr="_netBIOSNameServer[0]" key="data.dhcpv4Message.options[7].netBIOSNameServer[0]"/>
	<attrKeyMap attr="_optionId" key="data.dhcpv4Message.options[7].optionId"/>
	<attrKeyMap attr="_optionId" key="data.dhcpv4Message.options[8].optionId"/>
	<attrKeyMap attr="_netBIOSNodeType" key="data.dhcpv4Message.options[8].netBIOSNodeType"/>
	<attrKeyMap attr="_optionData" key="data.dhcpv4Message.options[9].optionData"/>
	<attrKeyMap attr="_optionId" key="data.dhcpv4Message.options[9].optionId"/>
	<attrKeyMap attr="_optionData" key="data.dhcpv4Message.options[10].optionData"/>
	<attrKeyMap attr="_optionId" key="data.dhcpv4Message.options[10].optionId"/>
	<attrKeyMap attr="_flags" key="data.dhcpv4Message.flags"/>
	<attrKeyMap attr="_hops" key="data.dhcpv4Message.hops"/>
	<attrKeyMap attr="_ciaddr" key="data.dhcpv4Message.ciaddr"/>
	<attrKeyMap attr="_hLength" key="data.dhcpv4Message.hLength"/>
	<attrKeyMap attr="_op" key="data.dhcpv4Message.op"/>
	<attrKeyMap attr="_schemaVersion" key="schemaVersion"/>
	<attrKeyMap attr="_http_version" key="headers.http_version"/>
	<attrKeyMap attr="_content_length" key="headers.content_length"/>
	<attrKeyMap attr="_content_type" key="headers.content_type"/>
	<attrKeyMap attr="_request_method" key="headers.request_method"/>
	<attrKeyMap attr="_http_user_agent" key="headers.http_user_agent"/>
	<attrKeyMap attr="_http_host" key="headers.http_host"/>
	<attrKeyMap attr="_http_accept" key="headers.http_accept"/>
	<attrKeyMap attr="_request_path" key="headers.request_path"/>
	<attrKeyMap attr="_accept_encoding" key="headers.accept_encoding"/>
	<attrKeyMap attr="_time" key="time"/>
	<attrKeyMap attr="_sourceId" key="sourceId"/>
	<attrKeyMap attr="_type" key="type"/>
	<attrKeyMap attr="_timePrecision" key="timePrecision"/>
	<attrKeyMap attr="_key" key="key"/>
	<attrKeyMap attr="_@version" key="@version"/>
	<attrKeyMap attr="_tags[0]" key="tags[0]"/>
	<attrKeyMap attr="_payloadType" key="payloadType"/>
	<attrKeyMap attr="_host" key="host"/>
	<attrKeyMap attr="_serverId" key="serverId"/>
	<attrKeyMap attr="_@timestamp" key="@timestamp"/>
</collectAndSetAttrByJSON>


<setEventAttribute attr="eventType">combineMsgId("BlueCat-JSON-", $_logId)</setEventAttribute>
<setEventAttribute attr="eventSeverity">1</setEventAttribute>

</parsingInstructions>

[Bruce7x2]

Dear cdurkin,

I copied and pasted the content you provided into my FortiSIEM environment. Although the parser validation passed, the test did not.

I directly copied the log from the “Test Event” section of your content into the test field.

In this regard, I would like to understand how we can troubleshoot when the test fails. Can we use the underlying logs of FortiSIEM for debugging? Or is there some small detail that I might have overlooked?
The Version of FortiSIEM : 7.1.3

[Bruce7x2]

I encountered a strange phenomenon in my environment. I tried using the default parser to run a test, such as the default CiscoUmbrellaJSONParser, but it resulted in an error. Normally, the default parser should pass the test, right?

[cdurkin_FTNT]

2 questions to answer here.
a) Where in the list (there is an order) did you place the new (BlueCat) parser?

b) The error you are reporting looks like you have two parsers enabled that match the test message.

ie: it was trying to match the parser you are testing with, but the message actually matched another parser which is also enabled.

I have attached the sample parser again, just in case there was a copy paste error.

There is a course available on the NSE that covers parsers.

[Bruce7x2]

Dear cdurkin,

a) Where in the list (there is an order) did you place the new (BlueCat) parser?
I don’t quite understand what you mean.
Basically, I created the parser according to the article “Ingesting JSON Formatted Events Received via HTTP(S) POST”.
…
Login to the Supervisor.
Navigate to ADMIN > Device Support > Parsers.
From the Search… field, enter PHCustomJsonParser.
Select it, and click Clone.
…

b) The error you are reporting looks like you have two parsers enabled that match the test message.
I suspect there is a bug with the “Test” button in my current FortiSIEM, otherwise, it wouldn’t be possible for me to fail the test even when using FortiSIEM’s built-in parser.
This is why I need to understand how to troubleshoot this issue using low-level commands.

Obviously, your environment can pass the test, and I believe the content you provided is correct.

[Rob_SIEM]

Hi Bruce,

Regarding parser order, when logs are matched against parsers, they are evaluated in order top to bottom on the parser page. If two parsers match the same test event, the error in your screenshot will occur. Typically if you have a modified parser that is a clone of the system parser, you should disable the system equiavalent, and only enable the clone.

The best thing would be to: 

1) Disable the cloned PHCustomJsonParser parser, enable only the system parser.

2) Click "New" under parsers tab

Parser Name: BlueCatAddrMgrParser_Custom

Device type: BlueCat AddressManager

Paste in the provided parser

Click Reformat

Click Validate

Click Test

Under Test events page -> Click New -> Paste a single test event, exactly as extracted from the FortiSIEM analytics page.

Click Test

After success -> click Back

Select the Enable checkbox on the prior page

Click Save

Lastly, click Apply at top of the parsers page, then apply all

Thanks,

[Rob_SIEM]

Hi Bruce,

As a final add, we are baking a parser into FortiSIEM for this log source, but here is the parser in its current form.

Click New under parser tab:

Same instructions as above, set the device type and parser name, and past the following file contents attached.

Use this event as the test event:

[PH_DEV_MON_CUSTOM_JSON]:[reptVendor]=BlueCat,[reptModel]=AddressManager,[reptDevName]=bluecatnetworks,[reptDevIpAddr]=141.193.213.21,[json]={"data":{"dhcpv4Message":{"chaddr":"00:00:00:00:00:01","siaddr":"0.0.0.0","secs":0,"hType":1,"xId":"0xc123456d","yiaddr":"192.168.1.1","giaddr":"192.168.1.2","options":[{"optionId":53,"messageType":"DHCP Ack","messageTypeId":5},{"serverId":"192.168.1.25","optionId":54},{"optionId":51,"leaseTime":67556},{"subnetMask":"255.255.255.0","optionId":1},{"router":["192.168.1.100"],"optionId":3},{"domainNameSever":["192.168.1.8","192.168.1.9","192.168.1.10"],"optionId":6},{"domainName":"domain.local","optionId":15},{"netBIOSNameServer":["192.168.1.82"],"optionId":44},{"optionId":46,"netBIOSNodeType":"H-node"},{"optionData":"00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00","optionId":81},{"optionData":"00 00 00 00 00 00 00 00 00 00 00 00 00 00 00","optionId":82}],"flags":"0x0000","hops":1,"ciaddr":"0.0.0.0","hLength":6,"op":2}},"schemaVersion":"1.0.0","headers":{"http_version":"HTTP/1.1","content_length":"37932","content_type":"application/json","request_method":"POST","http_user_agent":"Vector/0.22.1 (x86_64-unknown-linux-gnu)","http_host":"192.168.1.191:80","http_accept":null,"request_path":"/","accept_encoding":"identity"},"time":1719422428,"sourceId":"00000000-0000-0000-0000-000000000000","type":"dhcpactivity","timePrecision":"s","key":"00000000-0000-0000-0000-000000000000","@version":"1","tags":["_geoip_lookup_failure"],"payloadType":"dhcpv4-packet","host":"192.168.1.20","serverId":"dhcp-09","@timestamp":"2024-06-26T17:20:11.570Z"}

Thanks,