Unable to increase storage size of kafka and zookeeper pods

[shreyasarani23]

In my storage class I have given allowvolumeexpansion as true
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: kafka-storage-class
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Retain
allowVolumeExpansion: true
parameters:
storageaccounttype: Standard_LRS
kind: Managed
volumeBindingMode: WaitForFirstConsumer

And in my yaml file when I try to increase the size from 1Gi to 2Gi
it doesn't reflect in my pvc
kubectl describe pvc data-my-kafka-cluster-kafka-0 -n kafka
Name: data-my-kafka-cluster-kafka-0
Namespace: kafka
StorageClass: kafka-storage-class
Status: Bound
Volume: pvc-aa9f8636-5655-40d1-8a8a-51a59a2e76ef
Labels: app.kubernetes.io/instance=my-kafka-cluster
app.kubernetes.io/managed-by=strimzi-cluster-operator
app.kubernetes.io/name=kafka
app.kubernetes.io/part-of=strimzi-my-kafka-cluster
strimzi.io/cluster=my-kafka-cluster
strimzi.io/kind=Kafka
strimzi.io/name=my-kafka-cluster-kafka
Annotations: pv.kubernetes.io/bind-completed: yes
strimzi.io/delete-claim: true
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 1Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: my-kafka-cluster-kafka-0
Events:
please help

And how do I restart my kafka pods?

[scholzj]

Can you please format the YAML snippets as code? It will make it much more readable and easier to see any issues or errors. In which YAML do you increase the size? You have to do it in the Kafka custom resource. You should probably also share the Cluster Operator logs to better understand what is going on.

[shreyasarani23]

Hi @scholzj I did the following

1. I deleted my kafka cluster deployed. But my persistent volumes exists even if my cluster deleted.
2. Since I have 3 kafka and zookeeper pods I created 6 PVC's linking the PVCs to the appropriate PV and I changed storage to 2Gi
3. Next I edited my PV's by removing claimref section and changed storage:2Gi for all 6 PV's
4. Deployed my kafka cluster
5. when I do kubectl get pv I get this
   
6. when i do kubectl get pvc I get this
   
7. But now when I enter into pod by executing kubectl exec -it my-kafka-cluster-kafka-0 -n kafka -- /bin/bash and execute df -h
   Filesystem Size Used Avail Use% Mounted on
   overlay 124G 19G 106G 15% /
   tmpfs 64M 0 64M 0% /dev
   tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
   tmpfs 3.4G 52K 3.4G 1% /tmp
   shm 64M 0 64M 0% /dev/shm
   /dev/sda1 124G 19G 106G 15% /etc/hosts
   tmpfs 3.4G 0 3.4G 0% /var/opt/kafka
   tmpfs 3.4G 12K 3.4G 1% /opt/kafka/client-ca-certs
   tmpfs 3.4G 48K 3.4G 1% /opt/kafka/broker-certs
   tmpfs 3.4G 12K 3.4G 1% /opt/kafka/cluster-ca-certs
   /dev/sdd 976M 3.2M 957M 1% /var/lib/kafka/data

I can see that /dev/sdd is still 976M instead of 2Gi.
8. And also in azure portal it has not reflected the size of disk. Still it is showing 1Gi

I don't know why this happened. please tell me if there is any mistakes.
Please help me to resolve this issue

[scholzj]

I'm sorry ... I described the standard process of increasing the volume size on Kubernetes with Strimzi above, it is also in the docs. But I do not know anything about your infrastructure or cluster configuration. And your description does not really make it clear what exactly have you done and when.

[shreyasarani23]

I have followed this link https://strimzi.io/docs/operators/in-development/using.html#proc-resizing-persistent-volumes-str
but it didn't work for me it showed me the error that it cannot resize when the disk in attached state. So I did the above steps .
Can you give me the link where they have explained increasing volume size in kubernetes .

[shreyasarani23]

I was able to increase the size of my PV by following this link https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/. Thanks @scholzj for the help.

[scholzj]

Well, the Kubernetes link is exactly the same what the operator does. So I'm glad it helped but not sure it explains why did it not worked for you before.