View more than 10,000 line

[HafizJasmi]

Hi Guys,

Is it possible to view and export more than 10,000 line of data query. For example my customer request to provide log for 3 days, and the line if more than 10,000 is there other way to get all the data at once.

[RobertEvans]

Hi Muhammed,

A primary purpose of a SIEM is to facilitate searching of that data for interesting information to act on. If you exported 3 days of raw CSV data, what exactly are they doing with that data? Doing something like a raw text search from a text editor is not the way here. Sometimes it is better to inform clients that there is a more efficient way to search their data than whatever tool they are using. 

Did they state the request for why 3 days of logs are needed, or what they are looking for? It would be better to gather their requirements and run those searches from within the SIEM.

That being said, data export is limited to 100K records at at time due to volume. Many clients may not understand the sheer volume of events coming in at any one time. One of the SIEM's major purposes is to make reporting on that data simple, exporting the data in bulk for manual searching is inefficient. At 100 eps, that is 8,640,000 events per day. That expected traffic from a small branch firewall, now imagine you have 100 firewalls reporting into your SIEM, the event volume per day?

It simply doesn't make sense to export logs that are already searchable within the SIEM. I would recommend gathering their requirements and running their queries from within the SIEM.

If they are looking for compliance reporting, an auditor may ask for a sample data set on a certain day to prove log retention, but they don't need the whole day, only a sample of events during a given day.

Thanks,-------------------------------------------
Original Message:
Sent: Feb 28, 2021 08:33 PM
From: Muhammad Hafiz Safwan Bin Jasmi
Subject: View more than 10,000 line

Hi Guys,

Is it possible to view and export more than 10,000 line of data query. For example my customer request to provide log for 3 days, and the line if more than 10,000 is there other way to get all the data at once.

[HafizJasmi]

Thanks robert for the reply,

Yeah i know it does not make sense, yet it is hard to make the customer understand.

-------------------------------------------
Original Message:
Sent: Mar 01, 2021 07:40 AM
From: Robert Evans
Subject: View more than 10,000 line

Hi Muhammed,

A primary purpose of a SIEM is to facilitate searching of that data for interesting information to act on. If you exported 3 days of raw CSV data, what exactly are they doing with that data? Doing something like a raw text search from a text editor is not the way here. Sometimes it is better to inform clients that there is a more efficient way to search their data than whatever tool they are using. 

Did they state the request for why 3 days of logs are needed, or what they are looking for? It would be better to gather their requirements and run those searches from within the SIEM.

That being said, data export is limited to 100K records at at time due to volume. Many clients may not understand the sheer volume of events coming in at any one time. One of the SIEM's major purposes is to make reporting on that data simple, exporting the data in bulk for manual searching is inefficient. At 100 eps, that is 8,640,000 events per day. That expected traffic from a small branch firewall, now imagine you have 100 firewalls reporting into your SIEM, the event volume per day?

It simply doesn't make sense to export logs that are already searchable within the SIEM. I would recommend gathering their requirements and running their queries from within the SIEM.

If they are looking for compliance reporting, an auditor may ask for a sample data set on a certain day to prove log retention, but they don't need the whole day, only a sample of events during a given day.

Thanks,
Original Message:
Sent: Feb 28, 2021 08:33 PM
From: Muhammad Hafiz Safwan Bin Jasmi
Subject: View more than 10,000 line

Hi Guys,

Is it possible to view and export more than 10,000 line of data query. For example my customer request to provide log for 3 days, and the line if more than 10,000 is there other way to get all the data at once.