Error upgrade failed another operation install upgrade rollback is in progress

Hi @jlcrow.

We ask to please provide a test case or some way for the maintainers to test and reproduce the error you are experiencing. This way we can better help understand the issue and determine if there’s a fix we can apply to the next release. In most cases a simple set of steps to reproduce the issue on our end can help expedite the process.

Thanks.

@jlcrow Yikes. While I wouldn’t ask you to do this in production could you do an upgrade while using the --debug and -v 6 flags set. This will provide more diverse output that can help us find the problem.

@jlcrow Yikes. While I wouldn’t ask you to do this in production could you do an upgrade while using the --debug and -v 6 flags set. This will provide more diverse output that can help us find the problem.

Thanks @mattfarina, I’m pushing up a branch for deployment and will test with both versions of helm to see if I can reproduce

I have the same issues. I’ll try to get a debug output if possible.

I noticed this issue usually happens when you upgrade a helm chart with —wait and the upgrade clearly fails (like a crashloopbackoff or something like that) and helm is waiting until it reaches the timeout but the user does CTRL+C before reaching the timeout. After that I’ll get the same error as posted above:

STDERR:
  Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

I’m using helmfile, not helm directly, maybe its another problem with helmfile not sending the SIGTERM correctly.

Hi @bacongobbler, the described workaround did indeed works!

$ helm history kyc-api
REVISION    UPDATED                     STATUS          CHART               APP VERSION DESCRIPTION             
1           Mon Nov  9 14:57:36 2020    pending-install generic-base-0.2.1  0.1.0       Initial install underway

$ helm rollback kyc-api 1
Rollback was a success! Happy Helming!

 $ helm history kyc-api
REVISION    UPDATED                     STATUS          CHART               APP VERSION DESCRIPTION             
1           Mon Nov  9 14:57:36 2020    pending-install generic-base-0.2.1  0.1.0       Initial install underway
2           Mon Nov  9 15:06:15 2020    deployed        generic-base-0.2.1  0.1.0       Rollback to 1  

Looking at the werf/helm PR pretty much confirms that CTRL+C breaks the helm installation on 3.4.0.

Thanks for following up. Closing as a duplicate of #4558.

@bacongobbler I don’t understand how this was determined to be the same issue? There’s no ctrl+c running in my pipeline.

I still think there is a regression in 3.4.0 since I never had this issue before.

I’ve tried to reproduce this in a branch of my project and have been unable to at this point.

Okay. Have either of you been able to reproduce this issue?

I know that this particular error can occur when the previous release is in a PENDING_UPGRADE or some other transitional state — usually occurs due to a timeout or someone interrupting the upgrade midway through (hence why I linked to #4558). I have not heard of a regression in Helm 3.4.0 causing an upgrade to enter a transitional state other than through the ticket I linked earlier… Which isn’t isolated to just Helm 3.4.

@bacongobbler I haven’t had luck reproducing the issue, I’ve just bumped to 3.4.1 and upgraded the same deployment that previously failed under 3.4.0, will just assume the issue is resolved unless I see something else. Thanks for everything.

@bacongobbler I ran into this again this morning in one of our pipelines and did confirm that it appears to be a duplicate of https://github.com/helm/helm/issues/4558, it seems to be a bad timing issue where the pipeline kills the previous deployment while it’s in the middle of a deploy, I’m surprised we haven’t run into this before. I confirmed the workaround was successful, I ran a rollback to the last successful deployment in the history and then followed up with the new deployment and everything is back to normal.

Run ‘helm history —all’, job is probably pending, you’ll have to rollback to last successful deployment.

On Nov 29, 2020, at 5:39 PM, Victor Login notifications@github.com wrote:


my out:

helm upgrade shortlink-api ops/Helm/shortlink-api —install —wait —namespace=shortlink —set deploy.image.tag=0.7.0.16 —debug -v 6

history.go:53: [debug] getting history for release shortlink-api
upgrade.go:121: [debug] preparing upgrade for shortlink-api
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
helm.go:81: [debug] another operation (install/upgrade/rollback) is in progress
helm.sh/helm/v3/pkg/action.init
/home/circleci/helm.sh/helm/pkg/action/action.go:62
runtime.doInit
/usr/local/go/src/runtime/proc.go:5474
runtime.doInit
/usr/local/go/src/runtime/proc.go:5469
runtime.main
/usr/local/go/src/runtime/proc.go:190
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1373
UPGRADE FAILED
main.newUpgradeCmd.func2
/home/circleci/helm.sh/helm/cmd/helm/upgrade.go:156
github.com/spf13/cobra.(Command).execute
/go/pkg/mod/github.com/spf13/[email protected]/command.go:842
github.com/spf13/cobra.(Command).ExecuteC
/go/pkg/mod/github.com/spf13/[email protected]/command.go:950
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
main.main
/home/circleci/helm.sh/helm/cmd/helm/helm.go:80
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1373
helm ls —all

NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
GitLab CI Pipeline Job https://gitlab.com/shortlink-org/shortlink/-/jobs/879030237

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or unsubscribe.

why is this issue still closed? repro: on empty cluster

helm upgrade --install --version=3.13.0 --create-namespace --namespace ingress-nginx-2 --set controller.kind=DaemonSet --set controller.service.type=LoadBalancer --set controller.service.loadBalancerIP=127.0.0.1 ingress-nginx-2 ingress-nginx/ingress-nginx

and ^C it

I can reproduce with helm upgrade --install --atomic and interrupting it during the execution.
The second run will always return an error:

Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

Can be solved by:

helm rollback

I guess this proposal could solve this issue https://github.com/helm/helm/issues/8040

I also have the same issue described on this thread with v3.4.1

helm rollback can be a workaround for development machines, but unacceptable on production CI/CD pipelines

When the helm CLI receives a SIGTERM signal, it should exit gracefully leaving helm labels in a stable state, allowing further deployments without issues

The issue is not fixed yet and should be open again for further research

«fixed» by deleting all helm secrets

«fixed» by deleting all helm secrets

you don’t need to delete all helm secrets but only the last one. it sounds like a workaround but not like fix.

on v3.3.4 such case handled fine (see a picture attached). Using helm in GitLab CI and job cancelation became a problem after upgrading to 3.4.0.
v3.5.1 has the same issue too.

I am having the same issue without even the chart being initially present..

▶ helm3 ls -A | grep -i cert
▶ helm3 upgrade cert-manager --install --set installCRDs=true --namespace extra-services --version v1.1.0 jetstack/cert-manager
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

cert-manager chart (the one I am trying to install) does not even exist on my cluster.

Using

▶ helm3 version --short
v3.4.2+g23dd3af

I just tried it with

▶ helm3 version --short
v3.5.1+g32c2223

the end result is the same.

Why was this issue closed?

The OP determined his issue was a duplicate of #4558. As #4558 describes, there are a few cases where a helm upgrade can enter the PENDING_UPGRADE state in the event of a timeout. A helm rollback && helm upgrade resolves the issue; hence why it was closed as a duplicate of #4558 (the symptoms and the workaround is identical).

If you do not believe you are experiencing the same issue as the OP, please open a new ticket.

the problem is that helm rollback && helm upgrade is not a suitable solution for production deployments

Experienced this on Helm v3.5.2 caused by CTRL+C pressed during upgrade.
Workaround: kubectl delete secret sh.helm.release.v1.<RELEASE_NAME>.v<LATEST_REVISION>

I’ve seen this for helm installs too so no working history where one could rollback to.

Issue was dying k8s api so helm controller of flux2 was dying while installing the chart, what likely has the same effect as ctrl+c.

Had this issue because I tried to cancel a helm deployment from the command line. The workaround suggested by @Skaronator using a rollback got me past the error. Helm history looks like this now:

35              Thu Mar 18 10:09:15 2021        superseded      wordpress-0.1.6 5.4.2           Upgrade complete 
36              Thu Mar 18 10:32:16 2021        superseded      wordpress-0.1.6 5.4.2           Rollback to 34   
37              Thu Mar 18 10:42:48 2021        pending-upgrade wordpress-0.1.6 5.4.2           Preparing upgrade
38              Thu Mar 18 10:48:11 2021        superseded      wordpress-0.1.6 5.4.2           Rollback to 36   
39              Thu Mar 18 10:49:02 2021        deployed        wordpress-0.1.6 5.4.2           Upgrade complete 

answered Oct 16 ’22 18:10

Lucas Veljacic