Troubleshooting Tip: Troubleshooting FortiGateRugged with internal LTE modem showing SIM absent after WWAN failure

Description

This article describes how to diagnose and recover a FortiGate with an internal LTE modem that loses WWAN connectivity and reports SIM absent, blank IMEI/ICCID, or USB speed “pending” despite being detected.

Scope 

FortiGate, LTE.

Solution

FortiGates with a 5G SIM card and LTE modem support provide critical connectivity options for remote and backup networking. However, issues may arise that affect the functionality of these connections. Troubleshooting using the CLI can help diagnose and resolve such problems effectively.

Symptoms:

When one or more of the following symptoms are observed on the FortiGate:

• The WWAN interface is stuck in Connecting or Down.
• No IP address assigned to the WWAN interface.
• diagnose sys lte-modem modem-details shows:

diagnose sys lte-modem modem-details
LTE Modem detailed information:
system.global.airplane-mode: Off
Modem detected: Yes
Manufacturer:
Model:
Revision:
MSISDN:
ESN:
IMEI:
MEID:
Hardware revision:
Software revision:
SKU:
FSN:
PRL version: 0x0000
Modem Operation mode: QMI_DMS_OPERATING_MODE_ONLINE
Modem Operation USB speed: pending

The above output shows the manufacturer, model, IMEI, ICCID are blank and USB speed shows pending

• diagnose sys lte-modem sim-info shows:

diagnose sys lte-modem sim-info

LTE Modem SIM card information:
Active Slot: Slot 2
SIM state: QMI_UIM_CARD_STATE_ABSENT

• LTE debug logs shows:

diagnose debug reset
diagnose debug application lted -1
diagnose debug enable

read_lte_device(3053)-235: Can not read modem device 9 : timeout
modem_test_cmd(3137)-235: ####failed to read_modem_device fd:9
usb_sierra_dip_get_temp(1294)-235: Failed to run command: at!pctemp?
usb_lte_set_link_status(2453)-235: last link status:0, current link status:0
usb_lte_probe_device(2418)-235: v:1199, p:9091, ret:15
usb_lte_get_modem_dev(3229)-235: modem_dev:2
usb_lte_test_modem_cmd(3262)-235: cmd:|ati
|
open_lte_device(3015)-235: Trying to open modem device '/dev/ttyusb2'
modem_test_cmd(3128)-235: cmd:|ati
|, pid:235
read_lte_device(3053)-235: Can not read modem device 9 : timeout
modem_test_cmd(3137)-235: ####failed to read_modem_device fd:9
qmi_dev_read_msg(341)-235: Can not read QMI device 0 : timeout
usb_qmi_wwan_get_client_id(62)-235: Failed read msg(Invalid argument)
qmi_dev_read_msg(341)-235: Can not read QMI device 0 : timeout
usb_qmi_wwan_uim_read_transparent(2830)-235: Failed read msg(Invalid argument)
qmi_dev_read_msg(341)-235: Can not read QMI device 0 : timeout
usb_qmi_wwan_release_client_id(125)-235: Failed read msg(Invalid argument)

Try executing reboot on the FortiGate, but the LTE modem may reset, or reboot commands will return successful, but may not change the modem state. This is because the internal LTE modem is an embedded hardware device with its own firmware, SIM reader, and radio stack. In some conditions (such as carrier session failure, radio instability, firmware hang), the modem firmware enters a wedged state where the SIM/UIM interface stops responding, USB initialization does not complete, and FortiOS can detect the modem but cannot communicate with it. 

A FortiOS reboot (execute reboot) does not remove power from the LTE module, so the modem firmware remains stuck.
The resolution here is to perform a true power reset, as a software reboot may not be sufficient. This is done by powering off the FortiGate, disconnecting all power cables, and waiting about 2 minutes before reconnecting power and booting the device. This fully removes power from the LTE module and forces its firmware to reset. 

After booting, run the commands:

diagnose sys lte-modem modem-details
diagnose sys lte-modem sim-info

If IMEI and ICCID return and the SIM state becomes present, WWAN should reconnect and obtain an IP.
In conclusion, when an internal LTE modem reports SIM absent, blank IMEI/ICCID, or USB speed pending, the modem firmware is hung.
A full power drain is the correct recovery method.