Microsoft.AspNetCore.DataProtection.KeyManagement cache invalidation issue

[dougreesIG]

Hi there.

We use Microsoft.AspNetCore.DataProtection.KeyManagement for encryption purposes where multiple micro services rely on a shared keys.xml file held in blob storage that all services have access to.

This week we had an issue where one of the services created a new key unexpectedly over 2 weeks before the expiration date of the current key. it then started encrypting data with that new key. our other services didnt know anything about the new key due to the caching and we started getting failures.

It seems there is no way to invalidate the cache apart from modifying the keys by manually creating a new one, which is what our code does in these error cases. However, what that did was commit a new keys.xml file to the storage with another new key but the other recent key wasnt persisted I assume because when the second service committed its new key it did it directly from cache, meaning we lost the key from the first service.

we managed to restore the version that had the first of the two keys after some challenges so we resolved the situation. But I would like to know what should we be doing in this case? should we force a restart of the other micro services when a new key is created? how else can we force the cache to be reloaded apart from modifying the keys file?

Perhaps @amcasey you might have some ideas as I know you are familiar with this code.

Thanks.
Doug Rees

[amcasey]

Hey, @dougreesIG, thanks for reaching out!

Without much data to go on, my initial guess would be that the app had trouble connecting to your key repository (Azure?) and, having failed to do so, concluded that a new key was require. Unfortunately, there's no way for other instances to know when that happens so, as you observed, they tend to fail when they see data encrypted with it.

It seems there is no way to invalidate the cache apart from modifying the keys by manually creating a new one

I believe (but have not personally verified) that you could also use a dummy revocation - say revoking all keys created before 1900-01-01 - that should also invalidate the cache.

what that did was commit a new keys.xml file to the storage with another new key

This makes sense to me.

but the other recent key wasnt persisted

But this doesn't. Which instance generated the new key (to force cache invalidation)? Was it the same one that created the problematic key or a different one? Either way, I wouldn't expect a key to get lost (unless, I suppose, the generating instance lost access to the backing store and was unable to persist it).

we managed to restore the version that had the first of the two keys after some challenges so we resolved the situation

I think you mean you manually replaced keys.xml in the repository and each instance. That would suffice, but is definitely not something we want app authors to have to do.

Before I answer your other questions, it would help to know more about your app. Which version of aspnetcore are you running? Where are you storing your keys (i.e. your repository)? How are you protecting your keys?

Cheers,
Andrew

[dougreesIG]

Hi Andrew,

We are using .net 8 with Microsoft.AspNetCore.DataProtection.AzureKeyVault and AzureStorage versions 3.1.24.
We store the key in a blob storage account that is shared across multiple micro services.

The application has multiple micro services (will just refer to three of them here). Each shares the same keys file so that if one service encrypts, the others can decrypt.

Let me try and give a breakdown of what happened:

1. All services loaded up and pulled the keys file into their respective cache (lets call it key file v1 with current key expiring in around 2 weeks)

2. Service A decided for some reason to create a new key (file v2) around 2+ weeks before expiration of the current key.
   (see serviceA logs image attachment for read/write logs)

3. Service A used the new key (stored in the v2 file) to encrypt some data

4. Service B received the data with a key it didnt recognize due to its caching of v1 file, so it followed common guidance and generated a "dummy" key with a invalid start/end date (it did not do as you suggested here of revoking < 1900 keys). This generated a v3 version of the key file but only contained the v1 and invalid (dummy) key. it did not contain the v2 key from serviceA. So at this point the file lost the v2 key.

5. Anything from then on that was encrypted by the ServiceA using v2 key could then not be unencrypted by any other service.

6. We saw exceptions in our logs later in the day and took a while to figure out what happened, and managed to restore a snapshot of the v2 key file which contained both the v1 and v2 keys so we could then once again decrypt messages from ServiceA. We didnt care about the v3 file because that only had a dummy key plus the v1.

I think the problem is, when the ServiceB modified the key file (due to pushing the dummy key to invalidate the cache), it did so from memory cache meaning it just wrote out the file update from memory with the one new dummy key and the original v1 key. The issue is that the writing of the file needs to be done only after invalidating the cache and reloading from the file I think.

If we followed your suggestion of not creating a dummy key, but instead revoking all keys < 1900, would that also simply flush the cache from memory to the keys file? Or would that behave differently and fetch the current file first? Or does revoking the "fake" keys not in fact try and update the file, just invalidate the cache?

Many thanks for your guidance.

Regards,
Doug

[dougreesIG]

[amcasey]

I'll try to answer in more detail later, but my immediate reaction is that the azure provider has logic to prevent clobbering - basically, an instance should only be allowed to update the shared key file if it's (pre-update) local state matched the server state. So my guess would be that the instance that generated the bad key (service A) also failed to publish the bad key back to azure. If that's what happened, then B didn't clobber the bad key - it simply never existed in shared storage.

Do you have the option of testing out the 9.0 version of the data protection package(s)? We added some more resilience to server unavailability that we're hoping will prevent situations like this.

[amcasey]

Okay, I should be more specific. There's no way to reload the cache on all instances, even though that will ultimately be necessary.

On a single instance, you can just manually modify the cache by adding a key, setting a revocation, etc. The fact that that corrupted the store (by clobbering the bad key) is a bug.

Why isn't there just a "refresh the cache" API? I wasn't there, but my educated guess is that it should be possible but not convenient (which it is) because (a) it shouldn't be necessary and (b) refreshing the cache has a cost - potentially a very large one. Besides the extra load on the shared storage location, flushing the cache also impacts other requests being processed on the same instance. Since the client has the ability to force a refresh (just generate a random key that will be unknown to the server), there's a DoS risk. That may not apply in your scenario, in which case you can refresh instance by instance, as the need arises, but it will definitely result in some slow requests.

Again, I apologize that you have only half my attention right now - I'm on call - and I'll try to provide more details later.

[dougreesIG]

I very much appreciate your efforts to help anyway Andrew.

And yes we would be happy to handle the issue in each instance as the missing key is encountered, but from what we have seen modifying the cache by either adding a dummy/expired key, or revoking non-existent keys does indeed overwrite the stored file by dumping directly from the current memory cache, instead of perhaps reloading the file, then applying the change and rewriting it out.

[dougreesIG]

Ive just discovered we are using Microsoft.AspNetCore.DataProtection.AzureStorage nuget package which is deprecated in favour of Azure.Extensions.AspNetCore.DataProtection.Blobs

Perhaps that has something to do with it. I will look into this next.

[dougreesIG]

Yes I have a feeling this is it! Im looking at
https://github.com/Azure/azure-sdk-for-net/blob/Azure.Extensions.AspNetCore.DataProtection.Blobs_1.3.4/sdk/extensions/Azure.Extensions.AspNetCore.DataProtection.Blobs/src/AzureBlobXmlRepository.cs
and this seems to have logic to refresh first before overwriting. I will get that looked at first thing tomorrow by my devs.

[amcasey]

Woohoo! Glad we figured that out. I'll close this issue?