Attempted to process a blockchain with missing state root after CL restart in Gnosis

[kamilchodola]

Scenario:

Stop CL client for few minutes
Restart it and observe logs

Outcome:

Processing loop threw an exception. Block: 38060527 (0x0c6f49...3e6620), Exception: System.InvalidOperationException: Attempted to process a blockchain with missing state root 0xb4419585b059b7e174c2c07eda2e9766f3a87eb34f0753d784fb6f178cfef1d3
   at Nethermind.Consensus.Processing.BlockchainProcessor.PrepareBlocksToProcess(Block suggestedBlock, ProcessingOptions options, ProcessingBranch processingBranch) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 556
   at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer, String& error) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 363
   at Nethermind.Consensus.Processing.BlockchainProcessor.RunProcessingLoop() in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 286

Noticed that:

15 Jan 2025 23:07:19.953 Processed            38060523
15 Jan 2025 23:07:19.980 Processed            38060524
15 Jan 2025 23:07:19.992 Processed            38060525
15 Jan 2025 23:07:19.992 Processing loop threw an exception. Block: 38060527

Also block 526 was received during BeaconSync:

15 Jan 2025 23:07:17.228
 New beacon pivot: 38060526 (0x1042d1d24b13257ff9fa98efadbba456faf48e2a48a689b1e317a361db4c68d2)
15 Jan 2025 23:07:17.228
 Received ForkChoice: 0x1042d1...4c68d2, Safe: 0x3070fd...e572f8, Finalized: 0x83402b...554ddc
15 Jan 2025 23:07:17.228
 Syncing, Block 0x1042d1d24b13257ff9fa98efadbba456faf48e2a48a689b1e317a361db4c68d2 not found.
15 Jan 2025 23:07:16.501
 BeaconSync not finished - block 38060526 (0x1042d1...4c68d2) added to cache.
15 Jan 2025 23:07:16.501
 Received New Block:  38060526 (0x1042d1...4c68d2)      | limit    17,000,000    | Extra Data: Nethermind
15 Jan 2025 23:07:16.292
 BeaconSync not finished - block 38060525 (0x7011a9...c52846) added to cache.
15 Jan 2025 23:07:16.289
 Received New Block:  38060525 (0x7011a9...c52846)      | limit    17,000,000    | Extra Data: Nethermind
15 Jan 2025 23:07:13.426
 Beacon Headers from block 38060443 to block 38060525 |          0 /         83 | queue        0 | current       0 Blk/s  | skipped       0 Blk/s | total       0 Blk/s
15 Jan 2025 23:07:12.382
 Initialized beacon headers sync. lowestRequestedHeaderNumber: 38060525,lowestInsertedBlockHeader: 38060525 (0xb712865b166a8ea1a600253a468c8ee7cf06b6cd662ef04eb73b15e6caa17ac0), pivotNumber: 38060524, pivotDestination: 38060443
15 Jan 2025 23:07:12.240
 Start a new sync process, Request: ForkChoice: 0xb71286...a17ac0, Safe: 0x3070fd...e572f8, Finalized: 0x83402b...554ddc.
15 Jan 2025 23:07:12.240
 New beacon pivot: 38060525 (0xb712865b166a8ea1a600253a468c8ee7cf06b6cd662ef04eb73b15e6caa17ac0)

Seems like if we receive a new block before catching up with a chain head can cause us some missing.
Some debug logs for the case:

cutdebug.txt

[kamilchodola]

@brbrr why Medium? This is basically a crash which is not recoverable and is on Gnosis.

[brbrr]

I didn't understand that it's not recoverable. Feel free to update the priority to a proper one