companion pods goes to CrashLoopBackOff when deployment is restarted

[alvi-myalice-ai]

Initial checklist

• I understand this is a bug report and questions should be posted in the Community Forum
• I searched issues and couldn’t find anything (or linked relevant results below)

Link to runnable example

No response

Steps to reproduce

I have deployed companion in a gke kubernetes cluster following is my deployment, secrets and svc:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: companion
  namespace: companion
spec:
  minReadySeconds: 5
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: companion
  strategy:
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: companion
    spec:
      containers:
        - envFrom:
            - secretRef:
                name: companion-env
          image: docker.io/transloadit/companion:latest
          imagePullPolicy: IfNotPresent
          name: companion
          ports:
            - containerPort: 3020
              protocol: TCP
          resources:
            limits:
              memory: 150Mi
            requests:
              memory: 100Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /mnt/companion-data
              name: companion-data
            - mountPath: /mnt/data/
              name: companion-storage
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
        - emptyDir: {}
          name: companion-data
        - name: companion-storage
          persistentVolumeClaim:
            claimName: companion-data
---
apiVersion: v1
data:
  COMPANION_AWS_BUCKET: supersecretimaginarybucket
  COMPANION_AWS_KEY: h4)9\C#Z+pA6%%IbT8N#Ch
  COMPANION_AWS_REGION: ap-southeast-1
  COMPANION_AWS_SECRET: =D)£+u`|[B[Bbs709£\43{
  COMPANION_CLIENT_ORIGINS: frontend.mydomail.com
  COMPANION_DATADIR: /mnt/data/
  COMPANION_DOMAIN: https://mydomain.com/companion
  COMPANION_PROTOCOL: https
  COMPANION_REDIS_URL: redis://default:£32C9_q4hCb>M]@redis-master.companion.svc.cluster.local:6379
  COMPANION_SECRET: Z4f2*=ek%VPu@0p
kind: Secret
metadata:
  name: companion-env
  namespace: companion
type: Opaque
---
apiVersion: v1
kind: Service
metadata:
  name: companion
  namespace: companion
spec:
  ports:
    - port: 80
      targetPort: 3020
      protocol: TCP
  selector:
    app: companion

Expected behavior

The rollout restart deploy command should be able to schedule new pods without having to delete the service.

Actual behavior

When I run rollout restart deploy the new pods go into CrashLoopBackOff state with following in logs:

companion: 2025-01-12T06:05:07.675Z [warn] startup.secret auto-generating server preAuthSecret because none was specified
companion: 2025-01-12T06:05:07.684Z [error] startup.uploadUrls Running without uploadUrls is a security risk and Companion will refuse to start up when running in production (NODE_ENV=production)
companion: 2025-01-12T06:05:07.698Z [info] jobs.cleanup.start starting clean up job
companion: 2025-01-12T06:05:07.732Z [info]  Welcome to Companion! v5.4.1
companion: 2025-01-12T06:05:07.733Z [info]  Listening on http://localhost:tcp://34.118.239.236:80
node:events:492
      throw er; // Unhandled 'error' event
      ^

Error: listen EACCES: permission denied tcp://34.118.239.236:80
    at Server.setupListenHandle [as _listen2] (node:net:1734:21)
    at listenInCluster (node:net:1799:12)
    at Server.listen (node:net:1898:5)
    at Function.listen (/app/node_modules/express/lib/application.js:635:24)
    at Object.<anonymous> (/app/lib/standalone/start-server.js:11:22)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
Emitted 'error' event on WebSocketServer instance at:
    at Server.emit (node:events:514:28)
    at emitErrorNT (node:net:1778:8)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  code: 'EACCES',
  errno: -13,
  syscall: 'listen',
  address: 'tcp://34.118.239.236:80',
  port: -1
}

Node.js v18.17.1

where 34.118.239.236 seems to be the service IP. Unless svc is deleted this keeps happening.

[kvz]

Looks like you’re not allowed to listen on port 80.
Could it be you also need to pass a port environment variable?

[alvi-myalice-ai]

Looks like you’re not allowed to listen on port 80. Could it be you also need to pass a port environment variable?

I doubt it, the kubernetes service has the IP 34.118.239.236, when traffic is sent to 34.118.239.236:80 it forwards the traffic to one of the 2 pods of companion to port 3020. I followed the documentation provided here.

if you look at the error log:

Listening on http://localhost:tcp://34.118.239.236:80
node:events:492
      throw er; // Unhandled 'error' event
      ^

From what I understand the listen address should not be http://localhost:tcp://34.118.239.236:80

[kvz]

Not sure what is going on, but it looks suspicious to me that Companion thinks this is what it should open the port on:

Listening on http://localhost:tcp://34.118.239.236:80