Parse Fortigate Syslog to JSON with Regex works on 99 % of all logs - Need help with the last 1 %

[solo1]

I have log lines that I want to parse to JSON using Regex. I can now parse 99% of all logs, but the regex failes on a few log lines! I need help to complete the regex.

RegEx so far:

<(?P<priority>\d+)>(?P<timestamp>[A-Za-z]{3} \d{2} \d{2}:\d{2}:\d{2}) (?P<hostname>[^\s]+) date=(?P<date>\d{4}-\d{2}-\d{2}) time=(?P<time>\d{2}:\d{2}:\d{2}) devname=\"(?P<devname>[^\"]+)\" devid=\"(?P<devid>[^\"]+)\" eventtime=(?P<eventtime>\d+) tz=\"(?P<tz>[^\"]+)\" logid=\"(?P<logid>\d+)\" type=\"(?P<type>[^\"]+)\" subtype=\"(?P<subtype>[^\"]+)\" level=\"(?P<level>[^\"]+)\" vd=\"(?P<vd>[^\"]+)\" srcip=(?P<srcip>[^\s]+)(?: srcport=(?P<srcport>\d+))?(?: srcintf=\"(?P<srcintf>[^\"]+)\")?(?: srcintfrole=\"(?P<srcintfrole>[^\"]+)\")?(?: dstip=(?P<dstip>[^\s]+)(?: dstport=(?P<dstport>\d+))?)?(?: dstintf=\"(?P<dstintf>[^\"]+)\")?(?: dstintfrole=\"(?P<dstintfrole>[^\"]+)\")?(?: srcuuid=\"(?P<srcuuid>[^\"]+)\")?(?: dstuuid=\"(?P<dstuuid>[^\"]+)\")?(?: srccountry=\"(?P<srccountry>[^\"]+)\")?(?: dstcountry=\"(?P<dstcountry>[^\"]+)\")?(?: sessionid=(?P<sessionid>\d+))?(?: proto=(?P<proto>\d+))?(?: action=\"(?P<action>[^\"]+)\")?(?: policyid=(?P<policyid>\d+))?(?: policytype=\"(?P<policytype>[^\"]+)\")?(?: poluuid=\"(?P<poluuid>[^\"]+)\")?(?: policyname=\"(?P<policyname>[^\"]+)\")?(?: service=\"(?P<service>[^\"]+)\")?(?: trandisp=\"(?P<trandisp>[^\"]+)\")?(?: transip=(?P<transip>[^\s]+))?(?: transport=(?P<transport>\d+))?(?: duration=(?P<duration>\d+))?(?: sentbyte=(?P<sentbyte>\d+))?(?: rcvdbyte=(?P<rcvdbyte>\d+))?(?: sentpkt=(?P<sentpkt>\d+))?(?: rcvdpkt=(?P<rcvdpkt>\d+))?(?: appcat=\"(?P<appcat>[^\"]+)\")?

Example Log lines:

<189>Sep 23 10:18:17 cBBE9653E.static.as2116.net date=2024-09-23 time=12:18:17 devname="acdc-fortigate" devid="FGT40FTK2209B06Q" eventtime=1727086697110165699 tz="+0200" logid="0001000014" type="traffic" subtype="local" level="notice" vd="root" srcip=fe80::75d1:8573:acfa:2f2b srcport=5353 srcintf="wan" srcintfrole="wan" dstip=ff02::fb dstport=5353 dstintf="unknown-0" dstintfrole="undefined" replysrcintf="root" sessionid=3600841 proto=17 action="deny" policyid=0 policytype="local-in-policy6" service="udp/5353" trandisp="noop" app="udp/5353" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 rcvdpkt=0 appcat="unscanned" msg="Connection Failed"

<189>Sep 23 10:18:17 cBBE9653E.static.as2116.net date=2024-09-23 time=12:18:17 devname="acdc-fortigate" devid="FGT40FTK2209B06Q" eventtime=1727086696766758259 tz="+0200" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="root" srcip=192.168.1.110 srcport=45176 srcintf="lan" srcintfrole="lan" dstip=1.1.1.1 dstport=53 dstintf="wan" dstintfrole="wan" srcuuid="f8eef6a8-718a-51ee-c800-48fa677761f7" dstuuid="f8eef6a8-718a-51ee-c800-48fa677761f7" srccountry="Reserved" dstcountry="Australia" sessionid=7129435 proto=6 action="close" policyid=1 policytype="policy" poluuid="feafac0e-718a-51ee-3d8f-17868e4a5bab" policyname="Default test" service="DNS" trandisp="snat" transip=192.158.52.154 transport=45176 duration=2 sentbyte=426 rcvdbyte=408 sentpkt=6 rcvdpkt=5 appcat="unscanned"

<189>Sep 23 13:03:45 cBBE9653E.static.as2116.net date=2024-09-23 time=15:03:44 devname="acdc-fortigate" devid="FGT40FTK2209B06Q" eventtime=1727096624626748299 tz="+0200" logid="0001000014" type="traffic" subtype="local" level="notice" vd="root" identifier=0 srcintf="wan" srcintfrole="wan" dstip=ff02::fb dstintf="unknown-0" dstintfrole="undefined" replysrcintf="root" sessionid=3681215 proto=58 action="accept" policyid=0 policytype="local-in-policy6" service="icmp6/131/0" trandisp="noop" app="icmp6/131/0" duration=60 sentbyte=72 rcvdbyte=0 sentpkt=1 rcvdpkt=0 appcat="unscanned"

 
I manage to parse 98% with this regex, but some does not work, example of a line that does not work:

<189>Sep 23 13:14:27 cBBE9653E.static.as2116.net date=2024-09-23 time=15:14:27 devname="acdc-fortigate" devid="FGT40FTK2209B06Q" eventtime=1727097267439790399 tz="+0200" logid="0100040704" type="event" subtype="system" level="notice" vd="root" logdesc="System performance statistics" action="perf-stats" cpu=0 mem=45 totalsession=123 disk=0 bandwidth="205/273" setuprate=0 disklograte=0 fazlograte=0 freediskstorage=0 sysuptime=1664244 waninfo="name=wan,bytes=1680480093/39521483743,packets=3974237/367281389;" msg="Performance statistics: average CPU: 0, memory:  45, concurrent sessions:  123, setup-rate: 0"

I am missing the following three fields in my regex also:

* ReplySrcIntf:
* App:
* Msg:

(I am using LimaCharlie Adapter to fetch the logs):

sudo ./lc_adapter syslog client_options.identity.installation_key=1-2-3-4-5 client_options.identity.oid=6-7-8-9-10 client_options.platform=json client_options.hostname=dev-fw-fortigate40f-gcp client_options.sensor_seed_key=dev-fw-fortigate40f-gcp port=514 iface=127.0.0.1 is_udp=false "client_options.mapping.parsing_re=<(?P<priority>\d+)>(?P<timestamp>[A-Za-z]{3} \d{2} \d{2}:\d{2}:\d{2}) (?P<hostname>[^\s]+) date=(?P<date>\d{4}-\d{2}-\d{2}) time=(?P<time>\d{2}:\d{2}:\d{2}) devname=\"(?P<devname>[^\"]+)\" devid=\"(?P<devid>[^\"]+)\" eventtime=(?P<eventtime>\d+) tz=\"(?P<tz>[^\"]+)\" logid=\"(?P<logid>\d+)\" type=\"(?P<type>[^\"]+)\" subtype=\"(?P<subtype>[^\"]+)\" level=\"(?P<level>[^\"]+)\" vd=\"(?P<vd>[^\"]+)\" srcip=(?P<srcip>[^\s]+)(?: srcport=(?P<srcport>\d+))?(?: srcintf=\"(?P<srcintf>[^\"]+)\")?(?: srcintfrole=\"(?P<srcintfrole>[^\"]+)\")?(?: dstip=(?P<dstip>[^\s]+)(?: dstport=(?P<dstport>\d+))?)?(?: dstintf=\"(?P<dstintf>[^\"]+)\")?(?: dstintfrole=\"(?P<dstintfrole>[^\"]+)\")?(?: srcuuid=\"(?P<srcuuid>[^\"]+)\")?(?: dstuuid=\"(?P<dstuuid>[^\"]+)\")?(?: srccountry=\"(?P<srccountry>[^\"]+)\")?(?: dstcountry=\"(?P<dstcountry>[^\"]+)\")?(?: sessionid=(?P<sessionid>\d+))?(?: proto=(?P<proto>\d+))?(?: action=\"(?P<action>[^\"]+)\")?(?: policyid=(?P<policyid>\d+))?(?: policytype=\"(?P<policytype>[^\"]+)\")?(?: poluuid=\"(?P<poluuid>[^\"]+)\")?(?: policyname=\"(?P<policyname>[^\"]+)\")?(?: service=\"(?P<service>[^\"]+)\")?(?: trandisp=\"(?P<trandisp>[^\"]+)\")?(?: transip=(?P<transip>[^\s]+))?(?: transport=(?P<transport>\d+))?(?: duration=(?P<duration>\d+))?(?: sentbyte=(?P<sentbyte>\d+))?(?: rcvdbyte=(?P<rcvdbyte>\d+))?(?: sentpkt=(?P<sentpkt>\d+))?(?: rcvdpkt=(?P<rcvdpkt>\d+))?(?: appcat=\"(?P<appcat>[^\"]+)\")?"

[Jean-Philippe_P]

Hello solo1, 

Thank you for using the Community Forum. I will seek to get you an answer or help. We will reply to this thread with an update as soon as possible. 

Thanks, 

[Jean-Philippe_P]

Hello,

We are still looking for an answer to your question.

We will come back to you ASAP.

Thanks,

[Debbie_FTNT]

Dear solo1,

your Regex seems tailored to traffic logs, from what I can see, with strings like "srcintf" and "dstintf" that do not exist in event logs.

To get an idea of what fields the various logs can contain, you can use the Log Reference:
Event Logs:

https://docs.fortinet.com/document/fortigate/7.4.5/fortios-log-message-reference/398/event

Traffic Logs:

https://docs.fortinet.com/document/fortigate/7.4.5/fortios-log-message-reference/397/traffic

Are you trying to parse absolutely ANY log the FortiGate generates, or are you focussing on one particular log category (like only traffic logs)?

Cheers,

Debbie