has NO quorum, stalled on jetstream consumers

[1995parham]

We have the following log on one of the cluster nodes:

2022/06/22 20:30:25.131266 [WRN] JetStream cluster consumer '$G > rides > golchin-ride-cancel-passenger-new_ride' has NO quorum, stalled.

for one, of the consumers and this cause clients to get the following error:

nats: JetStream system temporarily unavailable

[derekcollison]

What does nats consumer info for that consumer report? What about nats stream info for the parent stream? And if you have access to the system account, nats server report jetstream.

Thanks..

[1995parham]

Yes nats strem info and nats server report jetstream was completely normal, but I got exact same error when I used nats consumer info.

nats: error: JetStream system temporarily unavailable (10008), try --help

[1995parham]

Also, this may be helpful. After restart, we got the following error:

Failed to install snapshot for '$G > rides > golchin-ride-cancel-passenger-new_ride' [C-R3M-Ha6B1HUu]: no message found

Consumer didn't have any issue, but didn't get any messages. Finally, everything backed to normal by deleting the PVCs. BTW, we are using the latest version.

[derekcollison]

What were you using for PVC? This running in K8S? Did you limit resources? Was the storage running low?

[1995parham]

The stream's storage is memory, so PVC is only there for as persistence storage. Yes, we are on k8s and there isn't any storage shortage. The resources are as below:

cpu: 1
memory: 4Gi

[derekcollison]

We recommend 4 cpu and at east 8Gi for JetStream enabled servers. And you said you are memory based, so how large is the stream?
And finally what server version are you running?

[1995parham]

Oh, we will increase resources then. We are on 2.8.4-alpine if I want to be specific. We limit the stream size to 1G and assign 2G to jetstream.

[derekcollison]

ok that should be ok then, still would give it a bit more cpu.

What inbound rate do you have to your stream?
For the consumer is it push or pull?

[1995parham]

We have 9k per second output and 7k per second input. Consumers are durable and has 3 replicas. Also, I have question, can we reduce these consumers' replicas?

[derekcollison]

Yes you can, and for those rates I would suggest raising CPU to at least 4.
For the consumers, ephemerals will default to R1 unless the stream has WorkQueue or Interest retention policies.
The server does allow for direct configuration now as well for consumer's R-factor and backing store.

How big are the messages?

[1995parham]

Messages isn't big (they are less than 100 KB). Can you point me to the documentation on Work Queue? They are consuming specific subjects (without using any wildcard). We have only 5 consumers on this cluster and all of them has 2 replicas besides their leader. I want to reduce the PVC usage as much as possible, so we can prevent issues like this one.

Also, are you suggesting having exactly 4 cores requested on k8s?

[derekcollison]

Can you share what nats stream info reports for your stream, and what nats consumer info reports for one of the consumers?

[jshiwamV]

We are facing the no quorum stalled issue, details are as follows:

• nats image version : nats:2.9.17-alpine
• No of nodes in K8's cluster - 3
• No of nats server pods - 5
• No of Streams - 4
• JetStream Replicas for Each Stream - 3
• Steps to reproduce:- Fail 2 nats server pods(nats-0 and nats-1) out of 5(i.e. nats-2,3,4 are running), One of the streams pops up with No quorom, stalled message
• Output of nats-3 pod: JetStream cluster stream '$G > MY_CONN' has NO quorum, stalled
• Stream Report

In the above image last row has all the pods down for that stream.

This conflicts with what documentation says,
https://docs.nats.io/running-a-nats-service/configuration/clustering/jetstream_clustering#cluster-size

[derekcollison]

If the stream is an R3, and you take two peers down from that set, the stream will not be able to make progress.

If you want to be able to survive 2 servers down, you would need R5, which is possible, but generally not recommended since cloud providers do things in 3s.

[jshiwamV]

Given a scenario where the stream is an R3 and supposedly 2 peers of that stream happens to be in a same ec2 instance node which goes down, there is no way to save that stream you mean. This sounds bothering. Any other suggestions or workarounds to prevent this apart from having R5?

[derekcollison]

Yes you can configure with placement tags such that the stream will select peers in different AZs..

[derekcollison]

https://docs.nats.io/nats-concepts/jetstream/streams#placement

[jshiwamV]

Thanks @derekcollison, this was helpful. Really appreciate it.