Fix Azure Event Hub events collection after upgrade to Python 3.9+

[lbahtarliev]

This post goes to the community members that collect events from Azure Event Hub. 

Maybe some of you have noticed, that after a certain version the AZURE PYTHON SDK method stopped working by default. 

THE PROBLEM: From Python 3.9+ Azure modules for Python moved from AMQP over Web Socket protocol to direct AMQP by default.

THE REAL PROBLEM: In enterprise, corporate and government alike, strictly controlled environments where Internet access is tightly controlled, access is usually organized via proxies and AMQP do not play well with these, not to speak the destination ports are 5671 (TLS) and 5672 (SASL - MANDATORY TLS UPGRADE. Consider it like STARTTLS specification of other protocols). For the inquisitive - more information here: https://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-security-v1.0-os.html#toc

THE SOLUTION: Аctually the solution is pretty easy once you've rolled up your sleeves and delved into the FortiSIEM backend where the magic happens The official Azure Python SDK supports specifying TransportType when initializing the EventHubConsumerClient. This allows to explicitly set the TransportType to AmqpOverWebsocket,thus making the events pulling go through Web Socket via port 443. And yes, HTTP/HTTPS proxies ARE supported. Welcome again peace of mind :)

As I cannot find an option to attach files to my post here in the community, here is a link to my Pastebin where you can find two .patch files, for the two backend python scripts used to discover event hub partitions and collect events from them: FortiSIEM Pastes 
The two patches inline:
getEventsFromEventHubPartition.py.patch

--- /opt/phoenix/bin/getEventsFromEventHubPartition.py  2024-12-07 06:57:40.000000000 +0200
+++ getEventsFromEventHubPartition.py   2025-01-10 16:55:04.317267699 +0200
@@ -8 +8 @@
-from azure.eventhub import EventHubConsumerClient,EventHubSharedKeyCredential
+from azure.eventhub import EventHubConsumerClient,EventHubSharedKeyCredential,TransportType
@@ -91 +91,2 @@
-       credential=credential)
+       credential=credential,
+       transport_type=TransportType.AmqpOverWebsocket)

getEventHubPartition.py.patch

--- /opt/phoenix/bin/getEventHubPartition.py    2024-12-07 06:57:40.000000000 +0200
+++ getEventHubPartition.py     2025-01-10 16:52:52.177139122 +0200
@@ -7 +7 @@
-from azure.eventhub import EventHubConsumerClient,EventHubSharedKeyCredential
+from azure.eventhub import EventHubConsumerClient,EventHubSharedKeyCredential,TransportType
@@ -55 +55,2 @@
-       credential=credential)
+       credential=credential,
+       transport_type=TransportType.AmqpOverWebsocket)

To apply them:

• Login to Super and/or Worker(s) via SSH
• Transfer both files - getEventHubPartition.py.patch and getEventsFromEventHubPartition.py.patch to a directory of your liking. I tend to create a directory under /opt for holding my customizations and tooling.
• Do a 'dry-run' to see if the patches will be successful:

  • > patch --dry-run --verbose --backup /opt/phoenix/bin/getEventHubPartition.py /opt/rnd/azureEventHub/getEventHubPartition.py.patch
    
    Hmm... Looks like a unified diff to me...
    The text leading up to this was:
    --------------------------
    |--- /opt/phoenix/bin/getEventHubPartition.py 2024-12-07 06:57:40.000000000 +0200
    |+++ /opt/rnd/azureEventHub/getEventHubPartition.py 2025-01-10 16:52:52.177139122 +0200
    --------------------------
    checking file /opt/phoenix/bin/getEventHubPartition.py
    Using Plan A...
    Hunk #1 succeeded at 7.
    Hunk #2 succeeded at 55.
    done

  • > patch --dry-run --verbose --backup /opt/phoenix/bin/getEventsFromEventHubPartition.py /opt/rnd/azureEventHub/getEventsFromEventHubPartition.py.patch
    
    Hmm... Looks like a unified diff to me...
    The text leading up to this was:
    --------------------------
    |--- /opt/phoenix/bin/getEventsFromEventHubPartition.py 2024-12-07 06:57:40.000000000 +0200
    |+++ /opt/rnd/azureEventHub/getEventsFromEventHubPartition.py 2025-01-10 16:55:04.317267699 +0200
    --------------------------
    checking file /opt/phoenix/bin/getEventsFromEventHubPartition.py
    Using Plan A...
    Hunk #1 succeeded at 8.
    Hunk #2 succeeded at 91.
    done

• If there are no errors displayed, you can proceed applying the patches. Just remove the '--dry-run' switch. KEEP the '--backup' switch should something goes wrong after all.

Congratulations! You made it through :)

THE REAL SOLUTION: FortiSIEM colleagues should consider the possibility to add an additional option in the 'Credentials' setting for 'Azure Event Hub'  to choose the TransportType. 

* WARNING: When upgrading these files get overwritten, thus the patches too. So you need to re-apply them.

* DISCLAIMER: This is not an official solution neither approved by Fortinet. It is a sole workaround I use in my environment at my own risk and acknowledgement. So use at your own risk.

FortiSIEM@FSM_FTNT