Kubernetes Helm stuck with an update in progress

I tried to run a Helm upgrade before running helm repo update and now it seems to be permanently stuck in "STATUS: pending-upgrade" and won't let me try to run the upgrade again.

Trying to run: helm upgrade --namespace coder --install --force --atomic --wait --version 1.13.2 --values ./coder.yaml coder coder/coder

outputs: Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

76543 次浏览
小开

I am facing the same. I used helm 3.4.1... It happens when the deployment is in pending and you use --atomic (which in helm3 implies also --wait).

I could not get upgrade working. Worst is that even helm -n code list did not show anything, so I could not do:

helm -n code code

As helm3 holds such info in secrets, just clean the respective secret(s) and do install (or upgrade --install , but without --atomic). In your case something like

helm delete --namespace code secret sh.helm.release.v1.code.v1

(where last v1 is the release number, so maybe list and delete all if you are ok with that).

and afterwars helm install.

NOTE: old objects (pods,etc) will be there, so the new install will try to merge things. It went OK for me, but note -> It's a HACK :)

More on: https://github.com/helm/helm/issues/5595

小开

TLDR: You need to rollback to another version first and then helm upgrade again:

helm rollback <release> <revision> --namespace <namespace>

This can happen for a few reasons, but it ultimately occurs when there's an interruption during the upgrade/install process. Commonly, you SIGKILL (Ctrl C) while the deployment is ongoing.

You'll notice that if you helm ls --namespace <namespace> while it's stuck in STATUS: pending-upgrade state, you'll see the following without any other information:

NAME    NAMESPACE   REVISION    UPDATED STATUS  CHART   APP VERSION

The best workaround currently is to rollback to another version, and then helm upgrade again:

helm rollback <release> <revision> --namespace <namespace>

revision is optional, but you should try to provide it.

more resources:

小开

In case is useful to someone, and in response to explicitsoul's comment, what fixed it to me was just:

helm delete <release> -n <namespace>

That removed the pending install (in my case, the first one so I hadn't a previous release to rollback to) and then I was able to run the install again.

What caused the stuck process in my case was a CTRL-C canceling the install command, so don't do that.

小开

This solution worked for me:

kubectl get secrets
kubectl delete secret sh.helm.release.v1.<RELEASE_NAME>.v<LATEST_REVISION>

Following the resolution described in this issue

小开

Here is what worked for me

  1. helm list --all This will list all the releases with their status
NAME  NAMESPACE       REVISION        UPDATED                                 STATUS               CHART                   APP VERSION
rel1  default         1               2021-06-04 14:15:37.652066 +0530 IST    deployed             rel1-3.32.0             0.46.0
rel2  default         29              2021-06-18 11:02:38.779801 +0530 IST    pending-upgrade      rel2-0.0.1
rel3  default         3               2021-06-17 11:27:14.608042 +0530 IST    deployed             rel3-0.0.1
  1. Notice that rel2 has status as pending-update . This happened because I did a Ctrl+C while upgrade was in progress
  2. All i had to do was rollback to previous revision in this case 28 helm rollback rel2 28 --namespace default
NAME   NAMESPACE       REVISION        UPDATED                                 STATUS          CHART          APP VERSION
rel1   default         1               2021-06-04 14:15:37.652066 +0530 IST    deployed        rel1-3.32.0    0.46.0
rel2   default         30              2021-06-18 11:26:07.555547 +0530 IST    deployed        rel2-0.0.1
rel3   default         3               2021-06-17 11:27:14.608042 +0530 IST    deployed        rel3-0.0.1
小开

kubectl get secrets kubectl delete secret sh.helm.release.v1.<RELEASE_NAME>.v<LATEST_REVISION>

By using the above command, it will remove the existing secrets in the middle of helm upgrade which will run and delete the stucked helm upgrade and it will generate the new one to proceed with the new upgrade.

小开

These are the steps that worked for me:

  1. See the status of your deployment (my was pending all the time)

    helm list --all

  2. Rollback to the previous version, for me was working already here, next step optional

    helm rollback <NAMESPACE_NAME> <Previous Version> --namespace <NAMESPACE_NAME>

  3. In case you want new/another deploy

    helm upgrade . . .

小开

In order to rollback to previous version you can just pass the release name:

helm rollback <RELEASE_NAME>

Where RELEASE_NAME can be seen when you run helm list --all -> under the NAME column.

(*) Add --namespace <namespace> if terminal context is not set to namespace.


提交答案