[FortiSIEM]How to Receive JSON Logs from Other Systems passively

[Bruce7x2]

Dear Team,

Based on the article "Ingesting JSON Formatted Events Received via HTTP(S) POST", I attempted to import logs from my DHCP Server (note: these are actual log files, not real-time logs sent by the DHCP Server). However, I encountered several issues, which are described below:

1.I'm currently using curl to send a JSON file (dhcpactivity.json) to FortiSIEM via HTTP (not HTTPS), including the -u '<admin>:<password>' information. However, the result returns a 401 Unauthorized error. What should I do about this? Or do I need to create a dedicated account in FortiSIEM (which I've actually done, but it still doesn't work)?
2.If I'm using HTTP, do I need to add the line "htpasswd -b /etc/httpd/accounts/passwds <user> '<password>'"? Or under what circumstances should this be added? (I've actually added this as well)
3.Before executing the curl command to send the logs, do I need to define Parsers in FortiSIEM first?
4.I noticed that the log example in the article (Ingesting JSON Formatted Events Received via HTTP(S) POST) is [PH_DEV_MON_CUSTOM_JSON]:[reptVendor]=<vendor>,[reptModel]=<model>,[reptDevName]=<reptName>,[reptDevIpAddr]=<reptIp>,[json]=<JSON>.
However, my log looks like the content shown below:

{"data":{"dhcpv4Message":{"chaddr":"00:00:00:00:00:01","siaddr":"0.0.0.0","secs":0,"hType":1,"xId":"0xc123456d","yiaddr":"192.168.1.1","giaddr":"192.168.1.2","options":[{"optionId":53,"messageType":"DHCP Ack","messageTypeId":5},{"serverId":"192.168.1.10","optionId":54},{"optionId":51,"leaseTime":67556},{"subnetMask":"255.255.255.0","optionId":1},{"router":["192.168.1.100"],"optionId":3},{"domainNameSever":["192.168.1.10","192.168.1.20","192.168.1.30"],"optionId":6},{"domainName":"domain.local","optionId":15},{"netBIOSNameServer":["192.168.1.40"],"optionId":44},{"optionId":46,"netBIOSNodeType":"H-node"},{"optionData":"00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00","optionId":81},{"optionData":"00 00 00 00 00 00 00 00 00 00 00 00 00 00 00","optionId":82}],"flags":"0x0000","hops":1,"ciaddr":"0.0.0.0","hLength":6,"op":2}},"schemaVersion":"1.0.0","headers":{"http_version":"HTTP/1.1","content_length":"37932","content_type":"application/json","request_method":"POST","http_user_agent":"Vector/0.22.1 (x86_64-unknown-linux-gnu)","http_host":"192.168.1.101:80","http_accept":null,"request_path":"/","accept_encoding":"identity"},"time":1719422428,"sourceId":"00000000-0000-0000-0000-000000000000","type":"dhcpactivity","timePrecision":"s","key":"00000000-0000-0000-0000-000000000000","@version":"1","tags":["_geoip_lookup_failure"],"payloadType":"dhcpv4-packet","host":"192.168.1.17","serverId":"dhcp-01","@timestamp":"2024-06-26T17:20:11.570Z"}
Can FortiSIEM parse this log content without modifying the Parser?

References:
[1] https://docs.fortinet.com/document/fortisiem/7.2.1/external-systems-configuration-guide/522142/inges...
[2] https://help.fortinet.com/fsiem/6-4-0/Online-Help/HTML5_Help/Ingesting_JSON.htm

[sioannou]

Hi @Bruce7x2 ,

Let me try to provide some answers: 

1) You have not created the appropriate account. Refer to your second question. You are required to create the account and setup the Authorization header (Basic "") in your POST request. 

2) Yes the account is required for ingesting the logs. This account should be utilised to send the logs.

4) When you perform the POST your URL should follow the structure https://<FSMNodeName>/rawupload?vendor=<vendor>&model=<model>&reptIp=<reptIp>&reptName=<reptHost>

Vendor, Model, Reporting IP and Reporting name are all extracted from the URI. 

3) For the custom Parser usually the answer is Yes. FortiSIEM has a basic parser for JSON ingested events that expects a single JSON Object with multiple Key, Value entries. I would suggest the following, start by sending the logs and check what the existing default Parser can parse. From there clone the parser and make any necessary modifications. 

Regards,

S

[Bruce7x2]

Dear Sioannou,
To be honest, I’m still unsure about how to define a Parser by observing the JSON format, and how they are interconnected. I don’t have any ideas up to this point.
Could you provide some valuable reference materials to help me attempt to define a Parser on my own?

[Bruce7x2]

Dear Team ,
I have transmitted the DHCP logs to FortiSIEM using curl with the account and password information. However, the system that generates these logs does not support features like curl, which can input account and password information as well as vendor, model, reptIp, and reptHost.
Given this, is there still a way for FortiSIEM to receive the logs?”

[sioannou]

Hi @Bruce7x2 ,

Does the system support any kind of log transmission?

S

[Bruce7x2]

Dear Sioannou,

no,it’s why i would like to know  whether fortisiem support http/json without account/password.

[Rob_SIEM]

Unfortunately it is required. Is this a Bluecat system? The Dhcpactivity logs with output type HTTP here? https://docs.bluecatnetworks.com/r/Address-Manager-Administration-Guide/Configuring-DHCP-Activity/9....

Looking at this log looks like a BlueCat DHCP Activity log, and their appliance seems to only support json webhooks, which is fine.

You can try this for the setup:

1) Make up a reporting ip/device name for the new log source for http web hook

#For reporting ip/name I just used the public site, it doesnt matter, just a CMDB entry showing where they came from. 

reptIp=141.193.213.21

reptName=bluecatnetworks

https://bluecatnetworks.com/

2) Create a Device type in SIEM

Login to Supervisor -> Admin tab -> Device Support -> Devices/Apps -> New 

Category: Application

Vendor: BlueCat

Model: AddressManager

Version: ANY

Device/App Group: Applications -> Infrastructure App -> DHCP Server

Biz Service Group: *skip for now

Access Protocol: 

Remove existing options (ssh,snmp,telnet)

Click Save

curl -k 'user:pw' https://<FSMNodeName>/rawupload?vendor=BlueCat&model=AddressManager&reptIp=141.193.213.21&reptName=bluecatnetworks

If following the BlueCat Guide:

https://docs.bluecatnetworks.com/r/Address-Manager-Administration-Guide/Configuring-DHCP-Activity/9....

Output Type: HTTP

Output URI: /rawupload?vendor=BlueCat&model=AddressManager&reptIp=141.193.213.21&reptName=bluecatnetworks

Bearer Token:

#Depending on how BlueCoat handles you can either type in

user:pw

OR

using a utility, compute the base64 of 'user:password' minus the single quotes. e.g. user:pw would be dXNlcjpwdw==

Disable healthcheck api

Host: https://<collectorIP>

#Full url example:

Full URL: https://<collectorIP>/rawupload?vendor=BlueCat&model=AddressManager&reptIp=141.193.213.21&reptName=bluecatnetworks

Resulting log should be something like this (truncated for brevity):

[PH_DEV_MON_CUSTOM_JSON]:[reptVendor]=BlueCat,[reptModel]=AddressManager,[reptDevName]=bluecatnetworks,[reptDevIpAddr]=141.193.213.21,[json]={"data":{"dhcp...}

Please test this out, and in FortiSIEM analytics, if you see the logs, export them to CSV format, and we can begin building a parser for them. 

[Bruce7x2]

Dear Rob,
You’re correct in your understanding, the log example is indeed from BlueCat.

I have successfully sent logs to FortiSIEM via "HTTP" using curl without the need for account and password information. This process involves adding a whitelist to /etc/httpd/conf/httpd.conf on FortiSIEM, which then allows me to send logs without adding account and password information to the curl command line.

Further, as you’re aware, if we want to use HTTPS and add a token in the form of a Bearer, BlueCat provides a Bearer Token.

How can I obtain this Bearer Token from FortiSIEM?

[Rob_SIEM]

The bearer token format is just "Basic + base64(username:pw)". When the sending product sends this, it puts it in the 'Authorization' header of the request. 

So the header -> value is Authorization -> Basic bXl1c2VybmFtZTpteXBhc3N3b3Jk

I've attempted to attach a python script that takes a user/pw and generates the resulting bearer token. Be sure to replace username/password with yours you generated when running htpasswd.

[Rob_SIEM]

Hi Bruce,

##Edit## 

After further review, my other reply is more accurate as this is a BlueCat AddressManager system, and does not seem to support FSM agents or access to utilities on linux like Rsyslog. Using the webhook features instead.

A few things to unpack here, but let me see if we can cover the existing feature for post of json events, future features coming up in the next month or so,

and some recommended alternate solutions.

Can you tell me what application is generating the log file? Is this on Linux or Windows?

1) Any custom log, or any log that is not natively ingested into FortiSIEM will require a FortiSIEM parser to understand.

   Even if you don't have a parser, they will be stored as unknown, but you cannot make use of rules/reports to filter on parsed fields.

  An added note on this, there can be many ways to ingest a log into FSM, so even if its supported by FSM, you have to use the supported way in case the log 

  structure changes depending on how you ship it and in what format - just something to keep in mind.

2) The feature here "Ingesting JSON Formatted Events Received via HTTP(S) POST" - https://help.fortinet.com/fsiem/6-4-0/Online-Help/HTML5_Help/Ingesting_JSON.htm

was a first cut feature to allow a third party application to send in json events to a FSM collector, but it is rigid in structure.

   Breaking it down:

This does not use any credentials defined in the FortiSIEM GUI, but apache credentials defined via SSH on the receiving collector:

SSH to the Collector and run the command: htpasswd -b /etc/httpd/accounts/passwds <user> <password>

1) SSH to collector using root

2) htpasswd -b /etc/httpd/accounts/passwds 'clientapp1' 'clientpass1234'

3) Using these creds will allow posting to the endpoint defined in the doc.

Example curl:

curl -kv -u 'clientapp1:clientpass1234' \ 

-d '{"logType": "someCustomLog", "data" : [] }' \

https://<FSMNodeName>/rawupload?vendor=<vendor>&model=<model>&reptIp=<reptIp>&reptName=<reptHost>

or if json is in a file

curl -kv -u 'clientapp1:clientpass1234' \ 

-d '@json_event.json' \

https://<FSMNodeName>/rawupload?vendor=<vendor>&model=<model>&reptIp=<reptIp>&reptName=<reptHost>

The tricky part is the url parameters here:

vendor and model = Must exactly match a device type defined in FSM gui under Admin -> Device Support -> Devices/Apps

e.g. Vendor = Fortinet, Model = FortiOS

reptIp = This allows CMDB object to be created, with your desired reporting ip (the device that sent the log)

reptName = Same as above but the hostname for the reporting device

A working guide using SAP Enterprise Threat Detection events is given here:

https://docs.fortinet.com/document/fortisiem/7.2.0/external-systems-configuration-guide/200971/sap-e...

When you post an event to this api, it generates an event like this in FSM (that a new parser must be built to handle)

[PH_DEV_MON_CUSTOM_JSON]:[reptVendor]=<vendor>,[reptModel]=<model>,[reptDevName]=<reptName>,[reptDevIpAddr]=<reptIp>,[json]=<JSON that you sent>

   This is a pretty advanced feature, and is limited to one type of log per api call which is rigid.

3) What I would recommend instead is to consider using an agent to collect this log file. Both the linux and windows agents support

tailing a log file (streaming new events coming in), and prepending a log header to them so they can be uniquely parsed.

If this is a linux system, and you do not have an agent, you can follow this guide below to setup logfile streaming via rsyslog. If you follow this route

be sure to update the TAG to something useful like "MyLogSource" to uniquely identify the log.

https://docs.fortinet.com/document/fortisiem/7.2.1/external-systems-configuration-guide/310791/apach...

Note: This guide is for Apache, but the procedure is the same for any log file on  section "Syslog via Rsyslog"

input(type="imfile" File="/var/log/httpd/ssl_error_log"

Tag="Apache_AccessLog:"

Severity="info"

Facility="local6")

Builds a log like this and ships it via syslog:

<179>Mar 22 00:41:50 lab1.example.com Apache_AccessLog: 192.0.20.0 - - [22/Mar/2022:00:41:48 +0000] "POST /phoenix/rest/h5/rt/start2?t=t1647909924028&s=333078424F54496950533135435470487275415A5974705451387635564B39496D4949717865776A HTTP/1.1" 200 36

4) If you use rsyslog, you can open a TAC case and ask for FSM team to build a parser (although it is not guaranteed or promised) , would need the following:

  a) The log source vendor/product

  b) Log source default file location

  c) Sample logs as exported from FortiSIEM in CSV format

5) A new feature coming in 7.2.2 of FortiSIEM

We realized the json post api above was somewhat rigid, so to support ease of export application scripts, we have a new api for collectors.

#You still need to setup apache user/pw using htpasswd

#Only available in FSM 7.2.2 and above

curl -kv -u 'clientapp1:clientpass1234' \ 

-d '@data.json' \

https://<FSMNodeName>/rawupload_mixed

The JSON format is different, there will be a detailed guide published on how to use it. The downside to this is an actual script needs to format

data into the desired json format below:

#Array of raw events

[

{

  "reptDevIpAddr" : "1.1.1.1",

  "phRecvTime" : 0123456789,

  "rawEventMsg" : "myCustomLog Here, or any log for that matter"

  },

{

  "reptDevIpAddr" : "3.3.3.3",

  "phRecvTime" : 0123456789,

  "rawEventMsg" : "some fortigate log"

  },   

]

The above data structure allows third party programs to collect the following data from a remote SIEM:

  1) The original sending log device IP

  2) The original time the log was sent to the original logging / siem product (to be preserved when parsed/imported into FSM)

  3) The raw event extracted from third party logging or siem solution

  4) Script bundles this up into the format above, and POSTs it to the API endpoint on the collector.

This is important because:

We can move a log event from remote siem application, preserve the original insert time, preserve the original logging device info, and force the raw event

to be parsed by our parser without modifying any parsers. 

6) Another new feature (webhooks)

  We will also support JSON webhooks from remote applications in any format such as Github/Gitlab hooks -> push to FSM collector. The benefit to this is no CLI work needed.

  More on this later.

TLDR; #3 and #4 items above are most likely the best way to get your events in, and if you can then show us the resulting log format in FSM we can most likely build you a parser.

Sorry for the wall of text here!

-Rob