JSON Log Parsing

[sioannou]

Hi all, 

Just picking the brains of the community to see if someone has found a better solution to the problem below: 

Problem: Receipt of a JSON log which contains an array of critical information. Like the one below.

{
 "id": 909999,
 "cstName": "test1",
 "status": "Normal",
 "nodes": [
    {
        "name": "node1",
        "nodeType": "type1",
        "status": "Critical",
        "fault": [
            {
                "attribute": "Hard-Drive",
                "value": "Critical",
                "reason": "Storage Limit Exceeded"
            },
            {
            "attribute": "CPU",
            "value": "Critical",
            "reason": "CPU Limit Exceeded"
            }
        ]
    }
 ]
}

Each JSON log can have 1-N number of Nodes and each node can have 1-10 fault codes.

How do you capture all Node details and all faults per node? 

Our Solution: 

The solution we are thinking right now is to utilise the parser function <when test="exist node[0]"> , <when test="exist node[1]">  etc... and then create multiple Event Type holders (i.e. node1, node2, node3, etc) and try to shoehorn the values back on the equivalent Event Type. 

The solution resembles more of "brute force" parsing :).

Has anyone come across this? Have you managed to find a more elegant solution to the problem above?

P.S. We have noticed within the PHCustomJsonParser there is function to split an array in multiple events, and this might solve the Node issue referenced above, what about the fault key above?

Thanks,

S

[MBerube]

I am having a very similar issue here. I have been reviewing the collectAndSetAttrByJsonArray function to extract the objects form the array.   To say the least, that function is not weel eplained in the documentation but it looks like it has other functionnality.

I would assume we would use the statement below but it doesn't work taht way.

<attrPosMap attr="field" pos="1"/>

[sioannou]

Hi @MBerube ,

I spend some time debugging the function to understand what is going on. So here is what I have learn. 

The collectAndSetAttrByJsonArray has the following format:

<collectAndSetAttrByJsonArray src="$_reason" sep=",">
</collectAndSetAttrByJsonArray>

 src=the Array you want to parse into the function. In the example I gave above this is the "fault" array. I collect that section by utilising the following before colling the collectAndSetAttrByJsonArray.

<collectAndSetAttrByJSON src="$_json">
<attrKeyMap attr="_reason" key="nodes.fault"/>
</collectAndSetAttrByJSON>

 sep = This is the array separator in most situations it will be a "," in some rare occasions this might be a space character. 

How do you match the elements:

To match the elements you need a unique value within each element of the array. For the example I gave above is the attribute section. Let's say I would like to match the CPU element of the array. I would use the code below to match the array element and then instruct the system to extract the value from the individual JSON keys. 

<collectAndSetAttrByJsonArray src="$_reason" sep=",">
<attrKeyMap attr="colCpuUtilState" key="entries.find(attribute='CPU', value)"/>
<attrKeyMap attr="colCpuUtilReason" key="entries.find(attribute='CPU', reason)"/>
</collectAndSetAttrByJsonArray>

The important section above is the key="entries.find(attribute='CPU', value)"

The "attribute" is the actual JSON key (name of the key) of the array that has the unique constrain value. In the above example the key name attribute (highlighted with bold below) changed to errorElement then the find function would change to  key="entries.find(errorElement='CPU', value)"

{
  "errorElement": "CPU",
  "value": "Critical",
  "reason": "CPU Limit Exceeded"
}

The second part of the find function "value" it is the JSON key you would like the data to be returned. In the above example the key name value (highlighted with bold below) changed to errorLevel, then the find function would change to key="entries.find(errorElement='CPU', errorLevel)"

{
  "errorElement": "CPU",
  "errorLevel": "Critical",
  "reason": "CPU Limit Exceeded"
}

So in general the entries.find finds the array element number based on a unique constrain and the second part of the function returns the key value indicated. It is exactly the same as "array[2].value" where the "array[2]" is determined dynamically by the find function. 

Hopefully the above makes sense. 

S

[FSM_FTNT]

Our event attributes typically stores a single value. Although we can save "3,6" to a string attribute, it will be problematic when we do aggregation query.

There's one way that when we see such events, we trigger more separate events to be generated.

In SAPEnterpriseThreatDetectionParser we have

<!-- input json, path, header, trailer, isDropBool -->
<setEventAttribute attr="_resultCount">splitJsonEvent($_json, "alerts", "[PH_DEV_MON_CUSTOM_JSON]:[reptVendor]=SAP,[reptModel]=ETD,SAP_Individual_Event,json=", "", "true")</setEventAttribute>

It means, if we parse $_json, we split by its "alerts" and trigger new events with the given header (3rd parameter) and tail (4th parameter). If we want to drop the original event, the last parameter is true.

E.g.

<setEventAttribute attr="_resultCount">splitJsonEvent($_json, "a.b", "HEAD", "TAIL", "true")

And we parse such _json from the original event:

{"a":{"b":[1,2,3]}}

Then the 2nd parameter "a.b" will position the array [1,2,3] and generate 3 extra events

HEAD1TAIL

HEAD2TAIL

HEAD3TAIL