Error script returned exit code 1

Make sure you have updated all relevant plugins, especially durable-task, as there have been diagnostic improvements. Typically the problem is that the Docker daemon does not share a filesystem with the client, which is a prerequisite for Image.inside to work.

Jesse Glick
added a comment — 2016-07-25 17:55 Make sure you have updated all relevant plugins, especially durable-task , as there have been diagnostic improvements. Typically the problem is that the Docker daemon does not share a filesystem with the client, which is a prerequisite for Image.inside to work.

I’m seeing a similar issue where the sh command inside a docker.image block always terminates after some delay. The point in the sh command that jenkins errors out varies which leads me to believe that jenkins is causing the problem. I’m on the latest version of jenkins and all plugins. Any tips on how to debug this?

Ben Mathews
added a comment — 2016-10-31 17:26 I’m seeing a similar issue where the sh command inside a docker.image block always terminates after some delay. The point in the sh command that jenkins errors out varies which leads me to believe that jenkins is causing the problem. I’m on the latest version of jenkins and all plugins. Any tips on how to debug this?

Typically it means workspace sharing between host & container is failing. If the agent itself is inside a container, make sure —volumes-from is included on the command line; if not, nothing will work.

Jesse Glick
added a comment — 2016-10-31 18:42 Typically it means workspace sharing between host & container is failing. If the agent itself is inside a container, make sure —volumes-from is included on the command line; if not, nothing will work.

The workspace is getting mounted (—volumes-from param is present) and is being interacted with. I’ve got a shell script that should take a couple minutes to complete. But within a couple seconds, the script will terminate and jenkins returns the above mentioned

ERROR: script returned exit code -1
Finished: FAILURE

The Jenkins log has the call stack

hudson.AbortException: script returned exit code -1
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:285)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:234)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Looking at the pipeline code in the call stack, it appears that Jenkins thinks my process is exiting. But (a) it never fails when I launch the process directly w/ «docker run …» and (b) it fails at a different spot every time.

I’ve replaced my script w/ «for i in `seq 1 20`; do echo $i;date;sleep 5;done» and it never fails. So, it is apparent that something in the interaction between jenkins and my script is failing.

FWIW, the script is a series of «python setup.py develop» commands.

Ben Mathews
added a comment — 2016-11-01 21:02 The workspace is getting mounted (—volumes-from param is present) and is being interacted with. I’ve got a shell script that should take a couple minutes to complete. But within a couple seconds, the script will terminate and jenkins returns the above mentioned

ERROR: script returned exit code -1
Finished: FAILURE

The Jenkins log has the call stack

hudson.AbortException: script returned exit code -1
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:285)
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:234)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang. Thread .run( Thread .java:745)

Looking at the pipeline code in the call stack, it appears that Jenkins thinks my process is exiting. But (a) it never fails when I launch the process directly w/ «docker run …» and (b) it fails at a different spot every time.

I’ve replaced my script w/ «for i in `seq 1 20`; do echo $i;date;sleep 5;done» and it never fails. So, it is apparent that something in the interaction between jenkins and my script is failing.

FWIW, the script is a series of «python setup.py develop» commands.

benm Hmm, does not sound like a familiar issue. The fake exit status -1 means that Jenkins cannot find the PID of the controller sh script which tracks the output and exit code of your actual script (also sh, unless you specified a #!/bin/command). The typical reason for the failure to find this PID when using Image.inside is that the container does not share the right mount with the agent, for example because —volumes-from was not passed when it should have been. But in your case it was, and you say other shell scripts work, so something trickier is happening. If you can narrow it down to a reproducible test case, that would help a lot of course. Otherwise you will need to inspect the process tree inside the container, to see if the wrapper script is really still running or not; and inspect the …job@tmp sibling workspace that holds the control directory with the PID, output, and exit code.

Jesse Glick
added a comment — 2016-11-01 21:49 benm Hmm, does not sound like a familiar issue. The fake exit status -1 means that Jenkins cannot find the PID of the controller sh script which tracks the output and exit code of your actual script (also sh , unless you specified a #!/bin/command ). The typical reason for the failure to find this PID when using Image.inside is that the container does not share the right mount with the agent, for example because —volumes-from was not passed when it should have been. But in your case it was, and you say other shell scripts work, so something trickier is happening. If you can narrow it down to a reproducible test case, that would help a lot of course. Otherwise you will need to inspect the process tree inside the container, to see if the wrapper script is really still running or not; and inspect the …job@tmp sibling workspace that holds the control directory with the PID, output, and exit code.

An update….

I cut the problem down to this repo

stage('run unit tests') {
      node() {
        docker.image("centos:6.7").inside() {
          sh 'for i in `seq 1 50`; do echo $i;date;sleep 2;done'
        }
}
}

This fails on both a docker using https://hub.docker.com/r/axltxl/jenkins-dood/ and jenkins installed on my ubuntu laptop via apt-get.

Before creating a ticket, I made one final test on our production jenkins server and it passed with flying colors four times consecutively. Obviously, something is strange with my box. Not sure what though. Any ideas would be appreciated. I’ll keep poking around

$ uname -a
Linux mathewslaptop 4.4.0-45-generic #66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="16.04.1 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.1 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
UBUNTU_CODENAME=xenial
$ docker -v
Docker version 1.12.1, build 23cf638

Ben Mathews
added a comment — 2016-11-03 17:11 An update….

I cut the problem down to this repo

stage( ‘run unit tests’ ) {
node() {
docker.image( «centos:6.7» ).inside() {
sh ‘ for i in `seq 1 50`; do echo $i;date;sleep 2;done’
}
}
}

This fails on both a docker using https://hub.docker.com/r/axltxl/jenkins-dood/ and jenkins installed on my ubuntu laptop via apt-get.

Before creating a ticket, I made one final test on our production jenkins server and it passed with flying colors four times consecutively. Obviously, something is strange with my box. Not sure what though. Any ideas would be appreciated. I’ll keep poking around

$ uname -a
Linux mathewslaptop 4.4.0-45- generic #66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/os-release
NAME= «Ubuntu»
VERSION= «16.04.1 LTS (Xenial Xerus)»
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME= «Ubuntu 16.04.1 LTS»
VERSION_ID= «16.04»
HOME_URL= «http: //www.ubuntu.com/»
SUPPORT_URL= «http: //help.ubuntu.com/»
BUG_REPORT_URL= «http: //bugs.launchpad.net/ubuntu/»
UBUNTU_CODENAME=xenial
$ docker -v
Docker version 1.12.1, build 23cf638

Are you using a (remote) agent on the production server but not in the test environment, or vice-versa? Generally should not matter except to the extent that it is the functioning of the docker command on the agent where the build is running which matters.

Jesse Glick
added a comment — 2016-11-08 18:01 Are you using a (remote) agent on the production server but not in the test environment, or vice-versa? Generally should not matter except to the extent that it is the functioning of the docker command on the agent where the build is running which matters.

The production server is running the builds on a centos 7 slave. My test environment is running jenkins in a docker container that is mounting /var/run/docker.sock from my laptop.

Ben Mathews
added a comment — 2016-12-24 16:36 The production server is running the builds on a centos 7 slave. My test environment is running jenkins in a docker container that is mounting /var/run/docker.sock from my laptop.

benm did you eventually solve this problem?  I’m seeing the same problem

where sh scripts are failing with -1.

I am also using:

docker.image(….).inside {

}

As a further note, my Jenkins instance is also a docker container.

Craig Rodrigues
added a comment — 2017-03-16 01:58 benm did you eventually solve this problem?  I’m seeing the same problem

where sh scripts are failing with -1.

I am also using:

docker.image(….).inside {

}

As a further note, my Jenkins instance is also a docker container.

Sorry, I should have followed up back in Dec. I’ve forgotten what happened with this. I stopped trying to do development locally, so I probably never resolved it.

Ben Mathews
added a comment — 2017-03-16 17:13 Sorry, I should have followed up back in Dec. I’ve forgotten what happened with this. I stopped trying to do development locally, so I probably never resolved it.

benm fair enough.  I’m running into the same problem so was just curious.

My understanding of the problem is that if your Jenkins server is running inside a Docker container, and then you try to start a docker container on the same server (so you are doing Docker container inside Docker container), then that doesn’t work so well.

The Durable Task plugin used by the pipeline **sh step has some complicated logic for how it figures out the process ID (pid) of the shell script that has been executed, and this logic gets confused when you do Docker inside Docker, and returns -1, even though your shell script is still running.  I ran into the -1 problems in JENKINS-32264 , (not in a Docker context).

Craig Rodrigues
added a comment — 2017-03-16 19:20 benm fair enough.  I’m running into the same problem so was just curious.

My understanding of the problem is that if your Jenkins server is running inside a Docker container, and then you try to start a docker container on the same server (so you are doing Docker container inside Docker container), then that doesn’t work so well.

The Durable Task plugin used by the pipeline **sh step has some complicated logic for how it figures out the process ID (pid) of the shell script that has been executed, and this logic gets confused when you do Docker inside Docker, and returns -1, even though your shell script is still running.  I ran into the -1 problems in  JENKINS-32264 , (not in a Docker context).

I have been having the same problems as benm with the shell script returning -1 if it is run inside a docker.inside block in a pipeline.

I took Ben’s testcase here: https://issues.jenkins-ci.org/browse/JENKINS-35370?focusedCommentId=275803&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-275803

and could reproduce the problem.

The environment I used to reproduce the problem was:

• host environment was a VM running Debian stretch
• docker version:Client:

Version:      1.13.1

API version:  1.26

Go version:   go1.7.5

Git commit:   092cba372

Built:        Wed Feb  8 06:44:30 2017

OS/Arch:      linux/amd64

Server:

Version:      1.13.1

API version:  1.26 (minimum version 1.12)

Go version:   go1.7.5
Git commit:   092cba372
Built:        Wed Feb  8 06:44:30 2017
OS/Arch:      linux/amd64

So the host environment, and the docker container are on the same machine and sharing the same file system.

Craig Rodrigues
added a comment — 2017-03-20 23:28 I have been having the same problems as benm with the shell script returning -1 if it is run inside a docker.inside block in a pipeline.

I took Ben’s testcase here: https://issues.jenkins-ci.org/browse/JENKINS-35370?focusedCommentId=275803&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-275803

and could reproduce the problem.

The environment I used to reproduce the problem was:

host environment was a VM running Debian stretch
docker version:Client:

Version:      1.13.1

API version:  1.26

Go version:   go1.7.5

Git commit:   092cba372

Built:        Wed Feb  8 06:44:30 2017

OS/Arch:      linux/amd64

Server:

Version:      1.13.1

API version:  1.26 (minimum version 1.12)

Go version:   go1.7.5
Git commit:   092cba372
Built:        Wed Feb  8 06:44:30 2017
OS/Arch:      linux/amd64

So the host environment, and the docker container are on the same machine and sharing the same file system.

I’m experiencing the same issue and it seems to depend on the docker image being used

I have a simple pipeline setup:

node('docker') {
    stage('test') {
        docker.image('r-base:3.4.0').inside() {
            sh(script: 'ping -c 2 jenkins.io')
        }
    }
}

which fails in all of about 100 test runs.

Log output:

Started by user Mathias Rühle
[Pipeline] node
Running on slave-2 in /home/jenkins/workspace/pipeline-test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] sh
[pipeline-test] Running shell script
+ docker inspect -f . r-base:3.4.0
.
[Pipeline] withDockerContainer
slave-2 does not seem to be running inside a container
$ docker run -t -d -u 1007:1007 -w /home/jenkins/workspace/pipeline-test -v /home/jenkins/workspace/pipeline-test:/home/jenkins/workspace/pipeline-test:rw -v /home/jenkins/workspace/pipeline-test@tmp:/home/jenkins/workspace/pipeline-test@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat r-base:3.4.0
[Pipeline] {
[Pipeline] sh
[pipeline-test] Running shell script
+ ping -c 2 jenkins.io
PING jenkins.io (140.211.15.101): 56 data bytes
64 bytes from 140.211.15.101: icmp_seq=0 ttl=44 time=163.801 ms
[Pipeline] }
$ docker stop --time=1 352151927bb3d4123f3bcfc467ab1a137b2829599a623807d22e18f9497cc742
$ docker rm -f 352151927bb3d4123f3bcfc467ab1a137b2829599a623807d22e18f9497cc742
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code -1
Finished: FAILURE

When I change the image version to 3.1.2:

node('docker') {
    stage('test') {
        docker.image('r-base:3.1.2').inside() {
            sh(script: 'ping -c 2 jenkins.io')
        }
    }
}

it succeeds in all of 20 test runs.

Log output:

Started by user Mathias Rühle
[Pipeline] node
Running on slave-2 in /home/jenkins/workspace/pipeline-test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] sh
[pipeline-test] Running shell script
+ docker inspect -f . r-base:3.1.2
.
[Pipeline] withDockerContainer
slave-2 does not seem to be running inside a container
$ docker run -t -d -u 1007:1007 -w /home/jenkins/workspace/pipeline-test -v /home/jenkins/workspace/pipeline-test:/home/jenkins/workspace/pipeline-test:rw -v /home/jenkins/workspace/pipeline-test@tmp:/home/jenkins/workspace/pipeline-test@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat r-base:3.1.2
[Pipeline] {
[Pipeline] sh
[pipeline-test] Running shell script
+ ping -c 2 jenkins.io
PING jenkins.io (140.211.15.101): 56 data bytes
64 bytes from 140.211.15.101: icmp_seq=0 ttl=44 time=163.756 ms
64 bytes from 140.211.15.101: icmp_seq=1 ttl=44 time=163.624 ms
--- jenkins.io ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 163.624/163.690/163.756/0.066 ms
[Pipeline] }
$ docker stop --time=1 443c8c93c17cc07d08c85be69304100b306191772ff4d5770537d1074b9d3679
$ docker rm -f 443c8c93c17cc07d08c85be69304100b306191772ff4d5770537d1074b9d3679
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

Another working docker image for me is perl:5.24.1.

I used 2 different setups. The first on my development maching running jenkins via

and having a docker container (customized imaged base on ubuntu:14.04) with ssh daemon as slave node. The slave node contains a statically linked docker executable (https://get.docker.com/builds/Linux/x86_64/docker-1.12.1.tgz) and the DOCKER_HOST environment variable is set to the docker network bridge ip 172.17.0.1 (the actual host machine). I would this consider a docker-in-docker setup.

The second setup is a jenkins instance running inside docker using the image jenkins:2.46.2. The jenkins slave is a separate vm running ubuntu 14.04 and having docker 1.12.1 installed. In my opinion this is not a docker-in-docker setup because of the jenkins slave being a real vm.

I get the same results on both setups.

Mathias Rühle
added a comment — 2017-05-11 12:04 — edited I’m experiencing the same issue and it seems to depend on the docker image being used

I have a simple pipeline setup:

node( ‘docker’ ) {
    stage( ‘test’ ) {
        docker.image( ‘r-base:3.4.0’ ).inside() {
            sh(script: ‘ping -c 2 jenkins.io’ )
        }
    }
}

which fails in all of about 100 test runs.

Log output:

Started by user Mathias Rühle
[Pipeline] node
Running on slave-2 in /home/jenkins/workspace/pipeline-test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] sh
[pipeline-test] Running shell script
+ docker inspect -f . r-base:3.4.0
.
[Pipeline] withDockerContainer
slave-2 does not seem to be running inside a container
$ docker run -t -d -u 1007:1007 -w /home/jenkins/workspace/pipeline-test -v /home/jenkins/workspace/pipeline-test:/home/jenkins/workspace/pipeline-test:rw -v /home/jenkins/workspace/pipeline-test@tmp:/home/jenkins/workspace/pipeline-test@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** —entrypoint cat r-base:3.4.0
[Pipeline] {
[Pipeline] sh
[pipeline-test] Running shell script
+ ping -c 2 jenkins.io
PING jenkins.io (140.211.15.101): 56 data bytes
64 bytes from 140.211.15.101: icmp_seq=0 ttl=44 time=163.801 ms
[Pipeline] }
$ docker stop —time=1 352151927bb3d4123f3bcfc467ab1a137b2829599a623807d22e18f9497cc742
$ docker rm -f 352151927bb3d4123f3bcfc467ab1a137b2829599a623807d22e18f9497cc742
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code -1
Finished: FAILURE

When I change the image version to 3.1.2:

node( ‘docker’ ) {
    stage( ‘test’ ) {
        docker.image( ‘r-base:3.1.2’ ).inside() {
            sh(script: ‘ping -c 2 jenkins.io’ )
        }
    }
}

it succeeds in all of 20 test runs.

Log output:

Started by user Mathias Rühle
[Pipeline] node
Running on slave-2 in /home/jenkins/workspace/pipeline-test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] sh
[pipeline-test] Running shell script
+ docker inspect -f . r-base:3.1.2
.
[Pipeline] withDockerContainer
slave-2 does not seem to be running inside a container
$ docker run -t -d -u 1007:1007 -w /home/jenkins/workspace/pipeline-test -v /home/jenkins/workspace/pipeline-test:/home/jenkins/workspace/pipeline-test:rw -v /home/jenkins/workspace/pipeline-test@tmp:/home/jenkins/workspace/pipeline-test@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** —entrypoint cat r-base:3.1.2
[Pipeline] {
[Pipeline] sh
[pipeline-test] Running shell script
+ ping -c 2 jenkins.io
PING jenkins.io (140.211.15.101): 56 data bytes
64 bytes from 140.211.15.101: icmp_seq=0 ttl=44 time=163.756 ms
64 bytes from 140.211.15.101: icmp_seq=1 ttl=44 time=163.624 ms
— jenkins.io ping statistics —
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 163.624/163.690/163.756/0.066 ms
[Pipeline] }
$ docker stop —time=1 443c8c93c17cc07d08c85be69304100b306191772ff4d5770537d1074b9d3679
$ docker rm -f 443c8c93c17cc07d08c85be69304100b306191772ff4d5770537d1074b9d3679
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

Another working docker image for me is  perl:5.24.1 .

I used 2 different setups. The first on my development maching running jenkins via

mvn hpi:run

and having a docker container (customized imaged base on ubuntu:14.04) with ssh daemon as slave node. The slave node contains a statically linked docker executable ( https://get.docker.com/builds/Linux/x86_64/docker-1.12.1.tgz) and the DOCKER_HOST environment variable is set to the docker network bridge ip 172.17.0.1 (the actual host machine). I would this consider a docker-in-docker setup.

The second setup is a jenkins instance running inside docker using the image jenkins:2.46.2 . The jenkins slave is a separate vm running ubuntu 14.04 and having docker 1.12.1 installed. In my opinion this is not a docker-in-docker setup because of the jenkins slave being a real vm.

I get the same results on both setups.

Encountered the same bug (at least, that is what I’m suspecting). By digging further and debugging the plugin into Eclipse, I was wondering if the docker plugin was faulty, or if it was not in the durable-task plugin instead, ending up my debugging onto the exitStatus() from ShellController class (see org.jenkinsci.plugins.durabletask.BournShellScript class).

Actually, I found out that this bug seemed to appear from our side after the recent Debian Stretch release, and suprisingly:

$ docker run -it --rm debian:stretch ps
docker: Error response from daemon: Container command 'ps' not found or does not exist..
$ docker run -it --rm debian:jessie ps 
PID TTY TIME CMD
1 ? 00:00:00 ps

The durable-task makes the assumption that it has the ps command to check if the process is still alive, which does not seem to be the case in new debian:stretch image. Note that if the sh command finishes before Jenkins has the need to control whether the process is still alive, then you won’t encounter the issue.

Pierre Mauduit
added a comment — 2017-06-29 12:00 Encountered the same bug (at least, that is what I’m suspecting). By digging further and debugging the plugin into Eclipse, I was wondering if the docker plugin was faulty, or if it was not in the durable-task plugin instead, ending up my debugging onto the exitStatus() from ShellController class (see org.jenkinsci.plugins.durabletask.BournShellScript class).

Actually, I found out that this bug seemed to appear from our side after the recent Debian Stretch release, and suprisingly:

$ docker run -it —rm debian:stretch ps
docker: Error response from daemon: Container command ‘ps’ not found or does not exist..
$ docker run -it —rm debian:jessie ps
PID TTY TIME CMD
1 ? 00:00:00 ps

The durable-task makes the assumption that it has the ps command to check if the process is still alive, which does not seem to be the case in new debian:stretch image. Note that if the sh command finishes before Jenkins has the need to control whether the process is still alive, then you won’t encounter the issue.

Something really weird (at least, I cannot explain): first clicking «scan repository now» before launching the actual build seems to solve my issue.

[edit]actually, it still failing after other build attempts

Pierre Mauduit
added a comment — 2017-07-17 16:09 — edited Something really weird (at least, I cannot explain): first clicking «scan repository now» before launching the actual build seems to solve my issue.

[edit] actually, it still failing after other build attempts

Hi,

I am using docker.image.inside() with a centos:7.3.1611 image for building maven project.

It was working perfect when my docker host was installed with Fedora 25.

I changed the docker host os from fedora to centos 7 and i started seeing this issue.

• The centos image has ps inside.
• The Jenkins server is running as docker container
• The pipeline is executed on a jenkins slave that runs on the same server

I can reproduce the issue with the example code from above:

node('build') {
    
   docker.image("builder-centos-7.3.1611-maven-3.5-jdk-8:2.2.0").inside() {
          sh 'for i in `seq 1 50`; do echo $i;date;sleep 2;done'
        }
    
}

The image above is based on cent os 7 but has jdk and maven installed.

Any hints? My whole environment is down right now…..

Pavel Georgiev
added a comment — 2017-08-10 08:00 — edited Hi,

I am using docker.image.inside() with a centos:7.3.1611 image for building maven project.

It was working perfect when my docker host was installed with Fedora 25 .

I changed the docker host os from fedora to centos 7 and i started seeing this issue.

The centos image has ps inside.
The Jenkins server is running as docker container
The pipeline is executed on a jenkins slave that runs on the same server

I can reproduce the issue with the example code from above:

node( ‘build’ ) {
    
   docker.image( «builder-centos-7.3.1611-maven-3.5-jdk-8:2.2.0» ).inside() {
          sh ‘ for i in `seq 1 50`; do echo $i;date;sleep 2;done’
        }
    
}

The image above is based on cent os 7 but has jdk and maven installed.

Any hints? My whole environment is down right now…..

My issue was resolved by either rebooting the server or making sure that the home folder of the user running the jenkins slave is exactly the same as the folder inside the centos image.

Hope that helps someone.

p.s.

The example code above also passes the command was executed 50 times without failure.

Pavel Georgiev
added a comment — 2017-08-10 09:44 My issue was resolved by either rebooting the server or making sure that the home folder of the user running the jenkins slave is exactly the same as the folder inside the centos image.

Hope that helps someone.

p.s.

The example code above also passes the command was executed 50 times without failure.

pgeorgiev, what do you mean by

home folder of the user running the jenkins slave is exactly the same as the folder inside the centos image

My setup is as follows:

node(‘alpine-node’) {
        docker.withServer('tcp://dockerhost:2375') {
            docker.image('centos-slave’).inside('--net=bridge') {
                sh '''
                    for i in $(seq 3); do sleep 1; echo $i; done
                '''
            }
        }  
    }
}

alpine-node is run from the same dockerhost as the centos-slave.

Should I map the home folder from dockerhost to the centos-slave?

Misha Yesiev
added a comment — 2017-08-17 14:01 pgeorgiev , what do you mean by
home folder of the user running the jenkins slave is exactly the same as the folder inside the centos image
My setup is as follows:

node(‘alpine-node’) {
        docker.withServer( ‘tcp: //dockerhost:2375’ ) {
            docker.image( ‘centos-slave’).inside(‘ —net=bridge’) {
                sh »’
                    for i in $(seq 3); do sleep 1; echo $i; done
                »’
            }
        }  
    }
}

alpine-node is run from the same dockerhost as the centos-slave .

Should I map the home folder from dockerhost to the centos-slave ?

Do not do such one-off mapping.

Probably this is a bug, possibly duplicate. The workaround is as always to just avoid inside. You can accomplish similar goals more portably and transparently using plain docker CLI commands.

Jesse Glick
added a comment — 2017-08-21 16:05 Do not do such one-off mapping.

Probably this is a bug, possibly duplicate. The workaround is as always to just avoid inside . You can accomplish similar goals more portably and transparently using plain docker CLI commands.

emishas thanks for the tip. After compiling the plugin with pr#40 Durable Task is able to keep track of the mvn process in my use case, and the pipeline jobs executes properly.

Bruno Didot
added a comment — 2017-08-23 20:44 emishas thanks for the tip. After compiling the plugin with pr#40 Durable Task is able to keep track of the mvn process in my use case, and the pipeline jobs executes properly.

jglick,

The workaround is as always to just avoid inside. You can accomplish similar goals more portably and transparently using plain docker CLI commands.

This does work when I just use ‘sh’ inside the .inside{}, then it would be like:

docker -H <dockerhost> exec <container> curl blablabla

But how do I execute other DSLs (like archiveArtifacts) inside the container?
Thanks!

Misha Yesiev
added a comment — 2017-09-07 18:25 jglick ,

The workaround is as always to just avoid inside. You can accomplish similar goals more portably and transparently using plain docker CLI commands.
This does work when I just use ‘sh’ inside the .inside{}, then it would be like:

docker -H <dockerhost> exec <container> curl blablabla

But how do I execute other DSLs (like archiveArtifacts) inside the container?
Thanks!

how do I execute other DSLs (like archiveArtifacts) inside the container?

You cannot, but you docker-cp files outside, etc.

Jesse Glick
added a comment — 2017-09-18 20:46 how do I execute other DSLs (like archiveArtifacts) inside the container?
You cannot, but you docker-cp files outside, etc.

I am encountering this same problem as well. I see there is at least one pull request associated with this issue emishas. Has the fix been released into an updated plugin somewhere?

Kevin Phillips
added a comment — 2017-09-20 12:35 I am encountering this same problem as well. I see there is at least one pull request associated with this issue emishas . Has the fix been released into an updated plugin somewhere?

Just to be clear, here are a few specifics of the behavior I’m seeing in our production environment in case it helps isolate the problem further:

• for some reason we only started experiencing this issue about 2 weeks ago. What’s even stranger is the instigating factor seems to be a server reboot of our Jenkins master instance. Based on my review, the agents remained unchanged during this outage, and the master remained the same (same core version, same plugin versions, same OS packages, same pipeline DSL code, etc.) but for some reason the reboot has caused this problem to start happening.
• For an example of the DSL code we’re using, see this issue I created — and resolved as a duplicate of this one: JENKINS-46969
• our environment is building a Docker image from a Dockerfile in the workspace and running the build operations inside a container launched from this image — which is slightly different than pulling an existing image from a Docker registry.
• our Docker image is based off a RHEL 7.2 base image
• I’ve confirmed our container has the ‘ps’ command line tool installed (there were some mentions that the Jenkins Docker APIs require this tool to be installed to work correctly)
• Using the exact same docker image and pipeline code results in successful builds about 75% of the time, so the failures are intermittent …. and yet frequent enough to be affecting production work
• We’ve reproduced the bug on 2 different Jenkins farms, one running core v2.43.3 and the other running v2.60.2. There are a variety of plugins installed on each of our farms, so if there are particular plugins that may play a part in this bug feel free to let me know and I’ll compile a list of the versions of those plugins for review.
• The agents attached to these masters are running one of the following host OSes: Centos 7.3, RHEL 7.3, RHEL 7.4
• The agents are running one of the following versions of Docker: 17.03.1-ce, 17.06.0-ce

As mentioned, these failures are affecting our production builds so any assistance with resolving them would be appreciated.

Kevin Phillips
added a comment — 2017-09-20 12:51 Just to be clear, here are a few specifics of the behavior I’m seeing in our production environment in case it helps isolate the problem further:

for some reason we only started experiencing this issue about 2 weeks ago. What’s even stranger is the instigating factor seems to be a server reboot of our Jenkins master instance. Based on my review, the agents remained unchanged during this outage, and the master remained the same (same core version, same plugin versions, same OS packages, same pipeline DSL code, etc.) but for some reason the reboot has caused this problem to start happening.
For an example of the DSL code we’re using, see this issue I created — and resolved as a duplicate of this one: JENKINS-46969
our environment is building a Docker image from a Dockerfile in the workspace and running the build operations inside a container launched from this image — which is slightly different than pulling an existing image from a Docker registry.
our Docker image is based off a RHEL 7.2 base image
I’ve confirmed our container has the ‘ps’ command line tool installed (there were some mentions that the Jenkins Docker APIs require this tool to be installed to work correctly)
Using the exact same docker image and pipeline code results in successful builds about 75% of the time, so the failures are intermittent …. and yet frequent enough to be affecting production work
We’ve reproduced the bug on 2 different Jenkins farms, one running core v2.43.3 and the other running v2.60.2. There are a variety of plugins installed on each of our farms, so if there are particular plugins that may play a part in this bug feel free to let me know and I’ll compile a list of the versions of those plugins for review.
The agents attached to these masters are running one of the following host OSes: Centos 7.3, RHEL 7.3, RHEL 7.4
The agents are running one of the following versions of Docker: 17.03.1-ce, 17.06.0-ce

As mentioned, these failures are affecting our production builds so any assistance with resolving them would be appreciated.

jglick I do appologize if this comment comes across as cynical, but telling people to just not use the built-in docker APIs provided by the Jenkins Pipeline infrastructure doesn’t seem very helpful to me. Cloudbees seems to have made it very clear that they are expecting everyone to adopt the new Pipeline subsystem as a new standard for Jenkins automation, and as part of that infrastructure are APIs for orchestrating Docker containers. Suggesting that these APIs are unstable and should simply not be used seems to contradict that stance. Further, as emishas has already pointed out, trying to get other build steps / plugins to interact correctly with a docker container managed in this way is going to be fragile at best, and impossible at worst. While this might be a reasonable workaround for the trivial case of running simple shell commands within the container, it does not seem to me to be a reasonable workaround for the general case.

Kevin Phillips
added a comment — 2017-09-20 13:00 jglick I do appologize if this comment comes across as cynical, but telling people to just not use the built-in docker APIs provided by the Jenkins Pipeline infrastructure doesn’t seem very helpful to me. Cloudbees seems to have made it very clear that they are expecting everyone to adopt the new Pipeline subsystem as a new standard for Jenkins automation, and as part of that infrastructure are APIs for orchestrating Docker containers. Suggesting that these APIs are unstable and should simply not be used seems to contradict that stance. Further, as emishas has already pointed out, trying to get other build steps / plugins to interact correctly with a docker container managed in this way is going to be fragile at best, and impossible at worst. While this might be a reasonable workaround for the trivial case of running simple shell commands within the container, it does not seem to me to be a reasonable workaround for the general case.

Is there any way to generate verbose output from Jenkins and / or Docker to help debug this issue more effectively? 

For example, there are some mentions above that Jenkins may be doing some ‘ps’ operations to detect whether the scripts running within the container have finished execution or not. Is there any way to get details as to what commands are being run, what their command line options are at the time they are run, what their stdout/stderr messages are, what their returns codes are, etc.? Similarly, if there are Docker tools being used to orchestrate these operations, is there any way to see which commands are being issued and when, and what their inputs/outputs are at runtime?

Based on my current evaluation, none of the system logs provide any sort of feedback in this regard. I’ve enabled verbose logging for Jenkins and Docker, examined their logs on both the master and the agents, I’ve looked as the sys logs, etc. and none of them give any indication of how or why the containers are being closed. In fact the only indication of any error happening at all is that message in the build log «ERROR: script returned exit code -1» which is misleading at best. It appears that the script — as in, the one being run as part of the ‘sh’ build step — isn’t actually returning that error code. Perhaps it’s an error code produced by a Docker command run by the Jenkins API under the hood. Not sure. Either way it appears to be of little to no help in debugging this problem.

Any suggestions on how to gather more intel on the problem would be appreciated.

Kevin Phillips
added a comment — 2017-09-20 13:42 Is there any way to generate verbose output from Jenkins and / or Docker to help debug this issue more effectively? 

For example, there are some mentions above that Jenkins may be doing some ‘ps’ operations to detect whether the scripts running within the container have finished execution or not. Is there any way to get details as to what commands are being run, what their command line options are at the time they are run, what their stdout/stderr messages are, what their returns codes are, etc.? Similarly, if there are Docker tools being used to orchestrate these operations, is there any way to see which commands are being issued and when, and what their inputs/outputs are at runtime?

Based on my current evaluation, none of the system logs provide any sort of feedback in this regard. I’ve enabled verbose logging for Jenkins and Docker, examined their logs on both the master and the agents, I’ve looked as the sys logs, etc. and none of them give any indication of how or why the containers are being closed. In fact the only indication of any error happening at all is that message in the build log «ERROR: script returned exit code -1» which is misleading at best. It appears that the script — as in, the one being run as part of the ‘sh’ build step — isn’t actually returning that error code. Perhaps it’s an error code produced by a Docker command run by the Jenkins API under the hood. Not sure. Either way it appears to be of little to no help in debugging this problem.

Any suggestions on how to gather more intel on the problem would be appreciated.

I had the same issue in my build where a Jenkins container, is running a Jenkinsfile which in turn runs commands inside another container, based on the latest ubuntu one, with the withDockerContainer or docker.image.inside command. A script that would take a few minutes checking out some git repositories would prematurely end with the ERROR: script returned exit code -1.

There is some discussion in Stackoverflow where it is mentioned that this issue is related with ps not installed in the container you want to run commands inside in. Indeed manually running my container with docker exec -it mycontainer bash, I noticed that running ps was failing with a command not found. I am not sure what the exact connection is but I can confirm that adding apt-get install -y procps in the Dockerfile of the container that I want to run commands inside, solved that issue for me, at least as a temporary workaround.

Ioannis Iosifidis
added a comment — 2017-10-24 15:19 I had the same issue in my build where a Jenkins container, is running a Jenkinsfile which in turn runs commands inside another container, based on the latest ubuntu one, with the withDockerContainer or docker.image.inside command. A script that would take a few minutes checking out some git repositories would prematurely end with the ERROR: script returned exit code -1.

There is some discussion in Stackoverflow where it is mentioned that this issue is related with ps not installed in the container you want to run commands inside in. Indeed manually running my container with docker exec -it mycontainer bash, I noticed that running ps was failing with a command not found. I am not sure what the exact connection is but I can confirm that adding apt-get install -y procps in the Dockerfile of the container that I want to run commands inside, solved that issue for me, at least as a temporary workaround.

I’m seeing this issue in Jenkins ver. 2.63.  I am running a python script inside a docker container and getting:

...
[Pipeline] stage
[Pipeline] { (Publish Classes)
[Pipeline] sh
[***] Running shell script
+ ./post_classes.py
...
ERROR: script returned exit code -1
Finished: FAILURE

I’ve tried to reproduce it with a simpler script, but I can’t.  It happens only with certain builds.

Update:

The docker container I was building didn’t contain ps (thank you iiosifidis for your message above). After adding ps, it fixed the issue.

Cosmin Stroe
added a comment — 2018-06-08 04:30 — edited I’m seeing this issue in Jenkins ver. 2.63.  I am running a python script inside a docker container and getting:

…
[Pipeline] stage
[Pipeline] { (Publish Classes)
[Pipeline] sh
[***] Running shell script
+ ./post_classes.py
…
ERROR: script returned exit code -1
Finished: FAILURE

I’ve tried to reproduce it with a simpler script, but I can’t.  It happens only with certain builds.

Update:

The docker container I was building didn’t contain ps (thank you iiosifidis for your message above). After adding ps, it fixed the issue.

I had the same issue，and I had checked my docker container，“ps” command can run in it，so I thought it had no relation with the cmmand “ps” installed in docker image，any other solutions？

wei lan
added a comment — 2018-09-26 06:38 I had the same issue，and I had checked my docker container，“ps” command can run in it，so I thought it had no relation with the cmmand “ps” installed in docker image，any other solutions？

5

윤태일
26 Ноя 2020 в 05:11

0

PDA
20 Авг 2020 в 08:02

0

Muhammad Tariq
26 Дек 2020 в 18:59

to jenkins…@googlegroups.com

Hi

I have a declarative script that runs a ‘bat’ step. That step builds some VS Studio projects on a Windows slave by calling msbuild:

            steps {

                bat »’

                    call «%VSMSBUILDCMD%»

                    msbuild %WORKSPACE%\MSVC\<proj> /p:Configuration=config_0 /p:Platform=»Win32″ /flp:logfile=Output.log /verbosity:normal

                    msbuild %WORKSPACE%\MSVC\<proj> /p:Configuration=config_1 /p:Platform=»Win32″ /flp:logfile=Output.log /verbosity:normal

                    msbuild %WORKSPACE%\MSVC\<proj> /p:Configuration=config_2 /p:Platform=»Win32″ /flp:logfile=Output.log /verbosity:normal

»’

}

The second msbuild command is failing (due to missing header files) and reports ‘Build Failed’, but the script step succeeds and the post stage executes the ‘success’ code.

Any idea why the step is not failing please?

Best regards

David

to jenkins…@googlegroups.com

A follow-up to this:

I ran the failing msbuild command from the command line (in the workspace) and checked the errorlevel, it was 1.

I then replayed the declarative script with some extra code:

msbuild %WORKSPACE%\MSVC\<proj> /p:Configuration=config_2 /p:Platform=»Win32″ /flp:logfile=ZodiacOutput.log /verbosity:normal

if errorlevel 1 (

   echo Failure Reason Given is %errorlevel%

   exit /b %errorlevel%

)

The Jenkins output was then:

Failure Reason Given is 1

Post stage

[Pipeline] script

[Pipeline] {

<snip>

[Pipeline] End of Pipeline

ERROR: script returned exit code 1

Finished: FAILURE

So I had to explicitly check the errorlevel to detect the failure. Does this mean there is this a bug in the declarative script plugin?

Best regards

David

to Jenkins Users

Where is «script» in the original example?

to Jenkins Users

>    bat »’

What type of quote(s) is that?  Three single quotes?

to jenkins…@googlegroups.com

Here’s a better view of my script:

pipeline {

    agent none

    options { buildDiscarder(logRotator(numToKeepStr:
’20’)) }

    triggers {

        pollSCM(‘H/5 * * * *’)

    }

    stages {

        stage(‘build_gcc’)
{

            <snip —
this stage runs on a
Linux node>

        }

        stage(‘build_VisualStudio’)
{

            agent { label
«jenkinswin10» }

            environment {

                VSMSBUILDCMD =
‘C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools\VsMSBuildCmd.bat’

            }

            steps {

                bat
»’

                    call «%VSMSBUILDCMD%»

                    msbuild %WORKSPACE%\MSVC\myproj.sln /p:Configuration=Release /p:Platform=»Win32″ /flp:logfile=ZodiacOutput.log
/verbosity:normal

                »’

            }

            post {

                <snip>

            }

        }

    }

}

to Jenkins Users

So you don’t mean «script», you mean using «bat» as you would «sh».   Is that right?

stage('Build') {
    steps {
        script {

to jenkins…@googlegroups.com

Yes, that’s right. Sorry for using wrong term.

to Jenkins Users

Take care batch error level only is valid for the last command and not a cumulative flag. So every msbuild command you run should check the error level result and exit batch according to it with proper error code. This will make the bat command failed properly. I personnaly would suggest, you run run each msbuild command into separated bat command so you will known which one failed, but that’s totally up to you. Also make sure you give the proper  bat argument returnStatus into your pipeline scripts.

to jenkins…@googlegroups.com

Thanks very much for your reply.

Yes, that was exactly the issue. I followed your advice and the script works correctly now.

I partly understand this. Where should I specify the returnStatus argument? (Note this is declarative).

to Jenkins Users

Not sure about declarative pipeline (I haven’t played with them yet). but for normal pipeline I use it this way:

returnStatus can be check or not if you want, if you don’t want to end the script for warnings like my above example. You can then use msbuild parser for example

to jenkins…@googlegroups.com

You received this message because you are subscribed to the Google Groups «Jenkins Users» group.
To unsubscribe from this group and stop receiving emails from it, send an email to
jenkinsci-use…@googlegroups.com.

Answers

• I’ve looped in the developers to take a look for other ideas.

  The caller code looks strictly at process.ExitCode from the child process:

        cmd.exe /C recreate_schema.bat WPRI1225 DQS SQL_Latin1_General_CP1_CI_AS

  When the Exit code is non-zero it logs that message. «Script process returned unexpected exit code: ‘1’.»

  The only other thing I can imagine is that the WaitForExit() call is happening but the cmd is slow to actually exit, some small timing condition if the computer is very slow (VM?) or if there is some 3rd party monitoring for process exits (antivirus, spyware
  tools?

  If you could see the cmd.exe command line in task manager, it may reveal something we haven’t seen (any rogue switch or something) but I am doubtful of that. I remember you said you couldn’t find the command line before. Did you click the button to show
  tasks from all users? I see they use an external cmd.exe process in the code, but maybe I’m wrong.

  Process Monitor should show us the Thread Exit and Process Exit events, and in the exit of cmd.exe we should be able to verify the exit code. Put an include filter on ProcessName=DQSInstaller.exe and ProcessName=CMD.exe

  Example of observing Process Exit in Process Monitor:

  

  By the way, are you clicking the icon to launch the DQSInstaller, or are you using an elevated Admin command prompt?

  Thanks, Jason

  ---

  Didn’t get enough help here? Submit a case with the Microsoft Customer Support team for deeper investigation — http://support.microsoft.com/select/default.aspx?target=assistance

  • Proposed as answer by
    JasonHowellMicrosoft employee
    Wednesday, February 29, 2012 12:15 AM
  • Marked as answer by
    _StevieC
    Wednesday, February 29, 2012 9:55 AM

• Hi Jason, I’ve fixed it!

  Using process monitor I was able to see that cmd.exe was exiting with an exit status of 1. So then I tried a bat file that didn’t do anything and that also had the same error of having an exit status of 1. By changing the path to have nothing in it and running
  process monitor again and with a little trial and error, I could see that the file cmd.exe was trying to open was c:tempansi32x64ansicon.exe. A quick search in the registry showed it at

  HKCUSystemMicrosoftCommand ProcessorAutoRun Reg_SZ: «c:tempansi32x64ansicon.exe» -p

  and that file doesn’t exist. I’ve removed the registry entry and now DQSInstaller.exe has completed successfully.

  I’m not sure what put that entry there and I don’t have that exe on my machine and I certainly didn’t install it. It’s all very strange.

  Thanks so much for your help in diagnosing what was wrong, you’ve been a great help and that was a great tip about looking at the exit code using process monitor.

  Thanks again,

  Steve.

  • Edited by
    _StevieC
    Tuesday, February 28, 2012 10:55 AM
  • Proposed as answer by
    JasonHowellMicrosoft employee
    Wednesday, February 29, 2012 12:15 AM
  • Marked as answer by
    _StevieC
    Wednesday, February 29, 2012 9:55 AM