FortiSIEM
FortiSIEM provides Security Information and Event Management (SIEM) and User and Entity Behavior Analytics (UEBA)
idabouzi
Staff
Staff
Article Id 407179
Description This article describes how to recover the etcd service after an upgrade failure
Scope FortiSIEM v7.3.0.
Solution

'etcd service' was first introduced in v7.3.0:

What's New in 7.3.0
https://etcd.io/docs/v3.3/faq/


The upgrade can fail because the 'etcd service' is not able to start. It could not fall back to the latest snapshot (snapshot file is missing or corrupt):


Jul 22 11:41:45 fortisiem bash[911808]: {"level":"warn","ts":"2025-07-22T11:41:45.034526+0200","caller":"snap/db.go:88","msg":"failed to find [SNAPSHOT-INDEX].snap.db","snapshot-index":7900079,"snapshot-file-path":"/var/lib/etcd/default.etcd/member/snap/0000000000788baf.snap.db","error":"snap: snapshot file doesn't exist"}
Jul 22 11:41:45 fortisiem bash[911808]: {"level":"panic","ts":"2025-07-22T11:41:45.034572+0200","caller":"etcdserver/server.go:518","msg":"failed to recover v3 backend from snapshot","error":"failed to find database snapshot file (snap: snapshot file doesn't exist)","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.NewServer\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:518\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\tgo.etcd.io/etcd/server/v3/embed/etcd.go:245\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcd\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:228\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:123\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:267"}
Jul 22 11:41:45 fortisiem bash[911808]: panic: failed to recover v3 backend from snapshot


Run the commands below to recover the 'etcd service':


sudo systemctl stop etcd
sudo rm -rf /var/lib/etcd/default.etcd
sudo systemctl start etcd


Run the upgrade again.

Contributors