KeyValuePairMultiValue within a JSON type log.

[Pascal]

Hello, 

I have an issue where we ingest a JSON type log which has multivalue key pairs. 

<132>1 2024-10-08T23:36:16.523272Z a347p001con61.itp.extra pam.rgt.gouv.qc.ca NILVALUE NILVALUE - {"adf":true,"significant":0,"udf":false,"virtualservice":"virtualservice-09330216-623b-4665-aebf-f6d69b96020c","report_timestamp":"2024-10-08T23:36:16.523272Z","service_engine":"z9900001awaf401-se-jmonv","vcpu_id":0,"log_id":661818,"client_ip":"3.98.92.14","client_src_port":48994,"client_dest_port":443,"client_rtt":9,"ssl_version":"TLSv1.2","ssl_cipher":"ECDHE-RSA-AES256-GCM-SHA384","http_version":"1.1","method":"GET","uri_path":"/note.txt","uri_query":"F_notini=&T_note=&nomentreprise=blah&filenote=../../winnt/win.ini","rewritten_uri_query":"F_notini=&T_note=&nomentreprise=blah&filenote=../../winnt/win.ini","user_agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)","persistence_used":true,"host":"142.80.132.13","persistent_session_id":3472328297252667977,"response_content_type":"text/html; charset=UTF-8","request_length":492,"cacheable":true,"pool":"pool-ae2724ab-84ce-4fae-9764-a36d138d199b","pool_name":"pam.rgt.gouv.qc.ca-qc-pool","server_ip":"10.246.60.11","server_name":"10.246.60.11","server_conn_src_ip":"10.246.77.19","server_dest_port":443,"server_src_port":6446,"server_rtt":2,"server_response_length":1027,"server_response_code":200,"server_response_time_first_byte":66,"server_response_time_last_byte":66,"response_length":1174,"response_code":200,"response_time_first_byte":66,"response_time_last_byte":66,"compression_percentage":0,"compression":"NO_COMPRESSION_CAN_BE_COMPRESSED","client_insights":"","request_headers":579,"response_headers":1033,"request_state":"AVI_HTTP_REQUEST_STATE_SEND_RESPONSE_BODY_TO_CLIENT","all_request_headers":{"Host":"142.80.132.13","Accept-Charset":"iso-8859-1,utf-8;q=0.9,*;q=0.1","Accept-Language":"en","Connection":"Keep-Alive","Cookie":"ORBSESS=udqarj8c1c30n2er45s3qmhosb808ddj8cmhnf4hhcd04pgcd7tdbmu50g2sf0i79i0e66jamnuipt9ips2ienucasvsvb1l","User-Agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)","Pragma":"no-cache","Accept":"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*"},"all_response_headers":{"Content-Type":"text/html; charset=UTF-8","Transfer-Encoding":"chunked","Connection":"keep-alive","Date":"Tue, 08 Oct 2024 23:36:16 GMT","Vary":"Accept-Encoding","Expires":"Thu, 19 Nov 1981 08:52:00 GMT","Cache-Control":"no-store, no-cache, must-revalidate","Pragma":"no-cache","Strict-Transport-Security":"max-age=63072000; includeSubdomains; preload","X-Frame-Options":"SAMEORIGIN","X-Content-Type-Options":"nosniff","X-XSS-Protection":"1; mode=block","Content-Security-Policy":"default-src 'self'; connect-src 'self' wss://142.80.132.13; script-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' https://assets.senhasegura.io data: chart.googleapis.com; style-src 'self' 'unsafe-inline'; font-src 'self' data: fonts.gstatic.com","Access-Control-Allow-Origin":"142.80.132.13"},"significant_log":["ADF_WAF_MATCH"],"headers_sent_to_server":{"X-Forwarded-For":"3.98.92.14","Host":"142.80.132.13","Accept-Charset":"iso-8859-1,utf-8;q=0.9,*;q=0.1","Accept-Language":"en","Cookie":"ORBSESS=udqarj8c1c30n2er45s3qmhosb808ddj8cmhnf4hhcd04pgcd7tdbmu50g2sf0i79i0e66jamnuipt9ips2ienucasvsvb1l","User-Agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)","Pragma":"no-cache","Accept":"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*","X-Forwarded-Proto":"https"},"headers_received_from_server":{"Date":"Tue, 08 Oct 2024 23:36:16 GMT","Content-Type":"text/html; charset=UTF-8","Transfer-Encoding":"chunked","Connection":"keep-alive","Vary":"Accept-Encoding","Expires":"Thu, 19 Nov 1981 08:52:00 GMT","Cache-Control":"no-store, no-cache, must-revalidate","Pragma":"no-cache","Strict-Transport-Security":"max-age=63072000; includeSubdomains; preload","X-Frame-Options":"SAMEORIGIN","X-Content-Type-Options":"nosniff","X-XSS-Protection":"1; mode=block","Content-Security-Policy":"default-src 'self'; connect-src 'self' wss://142.80.132.13; script-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' https://assets.senhasegura.io data: chart.googleapis.com; style-src 'self' 'unsafe-inline'; font-src 'self' data: fonts.gstatic.com","Access-Control-Allow-Origin":"142.80.132.13"},"server_ssl_session_id":"cbeff1995d917c00a2dbb2be50ddaffc","server_connection_reused":true,"server_ssl_session_reused":true,"vs_ip":"142.80.132.13","waf_log":{"rule_logs":[{"phase":"Request Body","rule_id":930100,"rule_group":"CRS_930_Application_Attack_LFI","msg":"Path Traversal Attack (/../)","matches":[{"match_element":"REQUEST_URI_RAW","match_value":"/note.txt?F_notini=&T_note=&nomentreprise=blah&filenote=../../winnt/win.ini","is_internal":false},{"match_element":"ARGS:filenote","match_value":"../../winnt/win.ini","is_internal":false}],"tags":["application-multi","language-multi","platform-multi","attack-lfi","paranoia-level/1","OWASP_CRS","CAPEC-255","CAPEC-153","CAPEC-126","CRS-group-930"],"rule_name":"Check for Path Traversal (1/2)","omitted_match_elements":0},{"phase":"Request Body","rule_id":930110,"rule_group":"CRS_930_Application_Attack_LFI","msg":"Path Traversal Attack (/../)","matches":[{"match_element":"REQUEST_URI","match_value":"/note.txt?F_notini=&T_note=&nomentreprise=blah&filenote=../../winnt/win.ini","is_internal":false},{"match_element":"REQUEST_URI","match_value":"/note.txt?f_notini=&t_note=&nomentreprise=blah&filenote=../../winnt/win.ini","is_internal":false},{"match_element":"ARGS:filenote","match_value":"../../winnt/win.ini","is_internal":false},{"match_element":"ARGS:filenote","match_value":"../../winnt/win.ini","is_internal":false}],"tags":["application-multi","language-multi","platform-multi","attack-lfi","paranoia-level/1","OWASP_CRS","CAPEC-255","CAPEC-153","CAPEC-126","CRS-group-930"],"rule_name":"Check for Path Traversal (2/2)","omitted_match_elements":0}],"status":"FLAGGED","latency_request_header_phase":104,"latency_request_body_phase":570,"latency_response_header_phase":22,"latency_response_body_phase":3,"rules_configured":true,"psm_configured":false,"application_rules_configured":false,"allowlist_configured":false,"allowlist_processed":false,"rules_processed":true,"psm_processed":false,"application_rules_processed":false,"memory_allocated":137056,"omitted_signature_stats":{"rules":0,"match_elements":0},"omitted_app_rule_stats":{"rules":0,"match_elements":0}},"request_id":"brp-TOpr-JVtf","servers_tried":1,"jwt_log":{"is_jwt_verified":false},"max_ingress_latency_fe":0,"avg_ingress_latency_fe":0,"conn_est_time_fe":8,"max_ingress_latency_be":0,"avg_ingress_latency_be":0,"conn_est_time_be":0,"source_ip":"3.98.92.14","vs_name":"pam.rgt.gouv.qc.ca"}

 The key value pairs : "rule_name" , "rule_id", "rule_group" and "msg" repeat themselves multiple times.  Unfortunately collectAndSetAttrByKeyValuePairMultiValue won't match JSON type attributes and I cannot use collectAndSetAttrByJSON as it does not support multi-value pairs. 

What would be my choice to capture these.  Unfortunately these logs are not structured therefore I can't use positions.  

- Pascal

[cdurkin_FTNT]

If you need to split the message into multiple messages you can look into the "splitJsonEvent" function.

https://help.fortinet.com/fsiem/6-6-4/Online-Help/HTML5_Help/paser-inbuilt-functions.htm#Split

Look at the "BitdefenderGravityZoneParser" for an example.

[Rob_SIEM · Accepted solution]

Hi Pascal,

Can you tell me a little about the log source / product api you are retrieving the logs from? I looked it over and this looks like a job we can achieve with a neat parser function called "splitJsonEvent"

https://help.fortinet.com/fsiem/6-6-4/Online-Help/HTML5_Help/paser-inbuilt-functions.htm#Split

See reference parser file: SAPEnterpriseThreatDetectionParser.xml

So the idea here is (based on initial look at the log above), you have a monolithic log that has many nested logs.

"waf_log": {
"rule_logs": [
{
"phase": "Request Body",
"rule_id": 930100,
"rule_group": "CRS_930_Application_Attack_LFI",
"msg": "Path Traversal Attack (/../)",
"matches": [
{
"match_element": "REQUEST_URI_RAW",
"match_value": "/note.txt?F_notini=&T_note=&nomentreprise=blah&filenote=../../winnt/win.ini",
"is_internal": false
},
{
"match_element": "ARGS:filenote",
"match_value": "../../winnt/win.ini",
"is_internal": false
}
],
"tags": [
"application-multi",
"language-multi",
"platform-multi",
"attack-lfi",
"paranoia-level/1",
"OWASP_CRS",
"CAPEC-255",
"CAPEC-153",
"CAPEC-126",
"CRS-group-930"
],
"rule_name": "Check for Path Traversal (1/2)",
"omitted_match_elements": 0
},

You have an array of waf rule logs. 

You could extract this json array to a variable like so:

<attrKeyMap attr="_ruleLogEvents" key="waf_log.rule_logs"/>

You could then have this function break up each array object as the body of a new log, and send that log to the parser module like a brand new smaller log.
<setEventAttribute attr="_resultCount">splitJsonEvent($_ruleLogEvents, "", "[PH_DEV_MON_CUSTOM_JSON]:[reptVendor]=MYTEST,[reptModel]=MYMODEL,Custom_Individual_Event,json=", "", "false")</setEventAttribute>

function args are:
input json array
json key pointing to the array if not at the top level, in our case ""
New log header string to prepend each of these events
Optional trailer string
true or false, indicator to drop the original monolithic event (false to keep both original event and split events)

The new events will fire one per object as:
[PH_DEV_MON_CUSTOM_JSON]:[reptVendor]=MYTEST,[reptModel]=MYMODEL,Custom_Individual_Event,json=<json body of a single rule_log array object>

You can then modify your parser to handle both the monolithic event, and the individual events, or have separate parsers for simplicity.

In the header string of the log, you can include anything you want from the monolithic log into the individual log header. 

This is just one way to handle this, but this an easy way to treat them as individual event objects for ease of parsing. 

[Pascal]

Thank you so much!  This worked perfectly, I have created a second parser to analyse the split log in order to facilitate readability. 

>>Can you tell me a little about the log source / product api you are retrieving the logs from?
These are logs coming from NSX WAF-AVI/Load Balancer.  Their Log page can be found here.

I really appreciate your help with all this.