Gitlab runner error checking for jobs forbidden

Runners are isolated machines (preferably not installed on the same machine as Gitlab) that pick up and execute CI/CD jobs for your Gitlab server.
A Runner can be specific to a certain project or serve any project in GitLab CI. A Runner that serves all projects is called a shared Runner.
You don’t have to have any runners at all if you are just using Gitlab for version control, but you will need at least one if you wish to implement CI/CD.
You can deploy as many runners as you want and it is a good idea to scale horizontally across multiple servers.

Related Posts

• GitLab Documentation Index

Steps

Install Docker

Deploy a gitlab-runner container with docker.

docker run -d 
  --name gitlab-runner 
  --restart always 
  -v $HOME/gitlab-runner-volume/config:/etc/gitlab-runner 
  -v /var/run/docker.sock:/var/run/docker.sock 
  gitlab/gitlab-runner:latest

This will result in any state being held in a directory at $HOME/gitlab-runner-volume. Feel free to change this path if you want it somewhere else, but I like it to be «in my face» so I don’t forget it’s there..

Update

To update your runner, pull the latest image, remove the existing container and re-deploy.

docker pull gitlab/gitlab-runner:latest
docker stop gitlab-runner && docker rm gitlab-runner

docker run -d 
  --name gitlab-runner 
  --restart always 
  -v $HOME/gitlab-runner-volume/config:/etc/gitlab-runner 
  -v /var/run/docker.sock:/var/run/docker.sock 
  gitlab/gitlab-runner:latest

Register The Runner

Go to the runners section of Gitlab and grab the url (4) and registration token (5)

Enter the container

After having grabbed the URL and token, you need to enter the container.

You now need to run a command inside the container to register the runner.
You can either explicitly specify your configuration options through arguments or run the command without the arguments and answer them in a series of interactive questions.
I recommend going with explicit if you are unsure.

Explicit Registration

Run the command below after having substituted the variables or having set them.
This uses the docker socket binding method so that the docker daemon is shared and your containers run directly on the host.
You can use other methods if you like.

REGISTRATION_TOKEN=""
URL=""

gitlab-runner register -n 
  --url $URL 
  --registration-token $REGISTRATION_TOKEN 
  --executor docker 
  --description "My Docker Runner" 
  --docker-image "docker:latest" 
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock

Interactive Registration

Run the following command (inside the container):

sudo gitlab-runner register

You will be asked a series of questions. When asked for the url and token, enter the url and token that you grabbed earler from Gitlab.

When asked which type of executor, I went with docker and I suggest you do the same unless you specifically know that you want to use a different executor such as Kubernetes.
You can read up more on runner executors.
Whilst shell is the easiest, it results in an environment that gets polluted from past runs, unlike using docker whereby you start with a clean slate each time.
I was unsure as to whether I wanted the docker executor or the docker-ssh executor, but it looks like the docker-ssh executor is being discontinued which made the choice easy.

Runner Registered

After having run the explicit command or using the interactive prompts, you will be shown the following error message:

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

If you navigate back to the runners section of Gitlab, you will see your newly registered runner.

Debugging

If you need to debug what it going wrong, use the command:

docker logs gitlab-runner

If you see this message:

ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory  builds=0

That means that you probably haven’t yet run the sudo gitlab-runner register command within your container yet. Please go back up to the register section.

At one point, I thought I needed to manually create the config.toml file using online examples, in order to register the runner.
This will only end up in pain as you need to pass the token to the gitlab-runner register and if you try using that token within the config file you will end up with messages like this in the logs:

ERROR: Checking for jobs... forbidden               runner=9f09eda9
ERROR: Checking for jobs... forbidden               runner=9f09eda9
ERROR: Checking for jobs... forbidden               runner=9f09eda9
ERROR: Runner https://gitlab.com/... is not healthy and will be disabled!

Running Out Of Space

If you don’t put something in place, your runners will probably eventually run out of space. Thus, I have it run a system prune every day.

sudo crontab -e

@daily /usr/bin/docker system prune -f

Even after that, my runner still ran out of space, so I added the following:

@daily /usr/bin/docker image prune --all --force

References

• Gitlab Docs — Run GitLab Runner in a container
• Gitlab runner blocked by IP
• Gitlab Documentation — Building Docker images with GitLab CI/CD

Last updated: 27th October 2022
First published: 16th August 2018

Now that the CI minutes are being gimped I need to have another go at setting up and running my own runners, which I failed at miserably in the past (I need docker-in-docker to work).

My docker-compose for the registrator and the runner:

version: "3.5"

services:
  runner:
    container_name: gl-runner
    image: gitlab/gitlab-runner:latest
    restart: unless-stopped
    volumes:
      - ./gitlab-runner.toml:/etc/gitlab-runner/config.toml
      - ./config:/etc/gitlab-runner
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - registrator

  registrator:
    container_name: gl-registrator
    image: gitlab/gitlab-runner:latest
    volumes:
      - ./config:/etc/gitlab-runner
    command: |
      register
      --non-interactive
      --executor "docker"
      --docker-image alpine:latest
      --url "https://gitlab.com/"
      --registration-token "<redacted>"
      --description "docker-runner"
      --tag-list "docker"
      --run-untagged="true"
      --locked="false"
      --access-level="not_protected"

The resulting config.toml:

[[runners]]
  name = "docker"
  url = "https://gitlab.com/"
  token = "<redacted>"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "alpine"
    privileged = true
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]

The registrator run just fine and the runner is registered to my group at gitlab.com, but when the runner starts right after the registrator it fails with this output:

❯ docker-compose up
Creating network "gitlab-runner_default" with the default driver
Creating gl-registrator ... done
Creating gl-runner      ... done
Attaching to gl-registrator, gl-runner
gl-registrator | Runtime platform                                    arch=amd64 os=linux pid=6 revision=738bbe5a version=13.3.1
gl-registrator | Running in system-mode.
gl-registrator |
gl-runner      | Runtime platform                                    arch=amd64 os=linux pid=6 revision=738bbe5a version=13.3.1
gl-runner      | Starting multi-runner from /etc/gitlab-runner/config.toml...  builds=0
gl-runner      | Running in system-mode.
gl-runner      |
gl-runner      | Configuration loaded                                builds=0
gl-runner      | listen_address not defined, metrics & debug endpoints disabled  builds=0
gl-runner      | [session_server].listen_address not defined, session endpoints disabled  builds=0
gl-registrator | Registering runner... succeeded                     runner=<redacted>
gl-registrator | Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
gl-runner      | ERROR: Checking for jobs... forbidden               runner=<redacted>
gl-registrator exited with code 0
gl-runner      | ERROR: Checking for jobs... forbidden               runner=<redacted>
gl-runner      | ERROR: Checking for jobs... forbidden               runner=<redacted>
gl-runner      | ERROR: Runner https://gitlab.com/<redacted> is not healthy and will be disabled!

What am I doing wrong here?

Пытаюсь запустить сборку через gitlab-runner docker:

$ gitlab-runner exec docker build:deb
ERRO[0000] Docker executor: prebuilt image helpers will be loaded from /var/lib/gitlab-runner. 
Running with gitlab-runner 11.2.0 (11.2.0)
Using Docker executor with image debian:buster ...
ERROR: Preparation failed: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: "gitlab-runner-cache": executable file not found in $PATH": unknown (executor_docker.go:412:0s)
Will be retried in 3s ...
Using Docker executor with image debian:buster ...
ERROR: Preparation failed: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: "gitlab-runner-cache": executable file not found in $PATH": unknown (executor_docker.go:412:0s)
Will be retried in 3s ...
Using Docker executor with image debian:buster ...
ERROR: Preparation failed: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: "gitlab-runner-cache": executable file not found in $PATH": unknown (executor_docker.go:412:0s)
Will be retried in 3s ...
ERROR: Job failed (system failure): Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: "gitlab-runner-cache": executable file not found in $PATH": unknown (executor_docker.go:412:0s)
FATAL: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: "gitlab-runner-cache": executable file not found in $PATH": unknown (executor_docker.go:412:0s)

Содержимое .gitlab-ci.yml:

stages:
  - build

build:deb:
  stage: build
  image: debian:buster
  tags:
  - deb
  before_script:
  - mkdir build && cd build
  - apt install cmake
  script:
  - cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=On -DRA_STATIC_LINK=ON ..
  - cmake --build . -- -j 8
  artifacts:
    paths:
    - build/run

Как исправить ошибку?