Gitlab 500 internal server error

I have a problem that I can not solve alone. I spent the day on it and I tried to find a solution by myself before creating this ticket. Context: I use the latest version of Gitlab proposed: same...

According to my notes a 500 appearing in the Gitlab-Runner doesn’t indicate much. It simply echoes the error it receives from workhorse which mangles the real message, some variant of 4XX from either gitaly or linguist into a 500. The first log to check is ones production.log but this seems to only log the 500 errors emitted by workhorse, so you have to got a level deeper and scan your workhorse.log.

Gitaly

Check the workhorse.log for a version mismatch between gitaly and workhorse. As I recall it was critical that both applications have comparable version numbers, there was a table one could check, as this dictated which protocols they understood.

Linguist

This was a really obscure issue I encountered. Essentially the version of Ruby used to run Gitaly and the version of Ruby used by Gitaly to run gitaly-ruby, the sub-processes it spawns internally, were different. This is apparently indicated by the cryptic messages 

time="2017-12-04T18:11:34+02:00" level=fatal msg="load config" config_path=/etc/gitaly/config.toml error="load linguist colors: exit status 1; stderr: "/usr/lib64/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/definition.rb:179:in `rescue in specs': Your bundle is locked to rake (12.1.0), but that version could not be found in any of the sources listed in your Gemfile. If you haven't changed sources, that means the author of rake (12.1.0) has removed it. You'll need to update your bundle to a different version of rake (12.1.0) that hasn't been removed in order to install. (Bundler::GemNotFound)\n\tfrom /usr/lib64/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/definition.rb:173:in `specs'\n\tfrom /usr/lib64/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/definition.rb:233:in `specs_for'\n\tfrom /usr/lib64/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/definition.rb:222:in `requested_specs'\n\tfrom /usr/lib64/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/runtime.rb:118:in `block in definition_method'\n\tfrom /usr/lib64/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/runtime.rb:19:in `setup'\n\tfrom /usr/lib64/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler.rb:99:in `setup'\n\tfrom /usr/lib64/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/setup.rb:20:in `<top (required)>'\n\tfrom /usr/lib64/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'\n\tfrom /usr/lib64/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'\n""
time="2017-12-04T18:17:54+02:00" level=info msg="Starting Gitaly" version="Gitaly, version 0.52.0, built 20171204.135804"

Note :

Please bear in mind that my notes are particular to Gentoo, a different version of GitlabHQ and may or may not be applicable to your situation accordingly. Please update your question as you find out more information since my notes may have further information that is relevant to your problem

Источник

Here is what happen, i got either a 500 or 502 error after an upgrade using Omnibus method. And got the error message «Gitlab is not responding»

40TPT

The first thing i did was to look for what could have happen. And the first place to see that is on the log file located at

/var/log/gitlab/gitlab-rails/production.log

And the log basically just give me an error as shown below,

Started GET "/users/sign_in" for 175.144.6.68 at 2015-12-12 23:48:54 +0800
Processing by SessionsController#new as HTML
Completed 500 Internal Server Error in 98ms (ActiveRecord: 10.8ms)

NoMethodError (undefined method `push_events=' for #<GitlabCiService:0x0000000463dba8>):
  app/models/project.rb:809:in `builds_enabled='
  app/controllers/application_controller.rb:194:in `add_gon_variables'

But when i do a status check, it gives me this

[[email protected] gitlab-rails]# gitlab-ctl status
run: gitlab-workhorse: (pid 4934) 1009s; run: log: (pid 4147) 1227s
run: logrotate: (pid 4942) 1008s; run: log: (pid 296) 3434s
run: nginx: (pid 4948) 1008s; run: log: (pid 299) 3434s
run: postgresql: (pid 4957) 1007s; run: log: (pid 301) 3434s
run: redis: (pid 4965) 1007s; run: log: (pid 294) 3434s
run: sidekiq: (pid 4972) 1005s; run: log: (pid 302) 3434s
run: unicorn: (pid 4990) 1004s; run: log: (pid 305) 3434s

I have pretty much no idea what is going on. But after trying out different ways, it seems to boil down to the following,

1. Check what is going on

Firing the following command should give you an idea what is going on with your configure.

sudo gitlab-rake gitlab:check

After that you could try see what is causing it.

2. Forget to turn on postgres before upgrade

Well, because gitlab said to shutdown gitlab before upgrading, hence i did this,

gitlab-ctl stop

which stops everything including postgres. Hence, database migration wasn’t possible. Therefore, i fire the following command and see whether that helps

gitlab-rake db:migrate

Now, after this i still got a 502 error but at least i’m not stuck with 500 error!

3. Forget to reconfigure after an upgrade

Well, if its not database migration, then every time you did a migration, remember to do a reconfigure!

gitlab-ctl reconfigure

Once i did this. Wait a while, and puff! The screen is back up!

Screen Shot 2015-12-13 at 12.00.47 AM

I’m just grateful everything is ok! Just remember to back up your VM image before doing all these upgrades!

Источник

Иногда возникает проблема при выполнении merge requests в gitlab.

В моем случае, у меня был старый gitlab, который пришлось обновить с версии 6.3 до версии 10.х (а в конечном итоге и до самой актуальной).

Ошибка имеет следующий вид:

Processing by Projects::MergeRequests::CreationsController#create as HTML

  Parameters: {«utf8»=>«<E2><9C><93>», «authenticity_token»=>«[FILTERED]», «merge_request»=>{«title»=>«Sonarqube Helm», «description»=>«», «assignee_ids»=>[«0»], «label_ids»=>[«»], «force_remove_source_branch»=>«0», «squash»=>«0», «lock_version»=>«0», «source_project_id»=>«332», «source_branch»=>«sonarqube», «target_project_id»=>«332», «target_branch»=>«master»}, «namespace_id»=>«roman», «project_id»=>«infrastructure»}

Completed 500 Internal Server Error in 2050ms (ActiveRecord: 132.4ms)

Started POST «/api/v4/jobs/request» for 192.168.1.2 at 2019—09—19 09:37:00 +0000

ActiveRecord::StatementInvalid (Mysql2::Error: Incorrect string value: ‘xF0x9Fx94x90mT…’ for column ‘diff’ at row 197:         INSERT INTO merge_request_diff_files (`diff`, `new_path`, `old_path`, `a_mode`, `b_mode`, `new_file`, `renamed_file`, `deleted_file`, `too_large`, `binary`, `merge_request_diff_id`, `relative_order`)

        VALUES (‘@@ -0,0 +1,23 @@n+# Patterns to ignore when building packages.n+# This supports shell glob matching, relative path matching, andn+# negation (prefixed with !). Only one pattern per line.n+.DS_Storen+# Common VCS dirsn+.git/n+.gitignoren+.bzr/n+.bzrignoren+.hg/n+.hgignoren+.svn/n+# Common backup filesn+*.swpn+*.bakn+*.tmpn+*~n+# Various IDEsn+.projectn+.idea/n+*.tmprojn+# OWNERS file for Kubernetesn+OWNERSn’, ‘helm/sonarqube/.helmignore’, ‘helm/sonarqube/.helmignore’, ‘0’, ‘100644’, 1, 0, 0, 0, 0, 5050, 0), (‘@@ —0,0 +1,19 @@n+apiVersion: v1n+name: sonarquben+description: Sonarqube is an open sourced code quality scanning tooln+version: 2.3.0n+appVersion: 7.9n+keywords:n+  — coveragen+  — securityn+  — coden+  — qualityn+home: https://www.sonarqube.org/n+icon: https://www.Started POST «/api/v4/jobs/request» for 192.168.1.2 at 2019—09—19 09:36:54 +0000

Причина кроется в размере выделяемой памяти для той или иной кодировки https://dev.mysql.com/doc/refman/5.5/en/charset-charsets.html

Проверять необходимо всю базу, но начать можно с табличек в базе gitlabhq_production с префиксами «merge_»:

merge_request_diff_commits

merge_request_diff_files

merge_request_diffs

merge_request_metrics

merge_requests

merge_requests_closing_issues

Кодировка данных таблиц была latin-swedish-ci, после конвертации в utf8mb4_general_ci проблемы с merge requests были исправлены.

Выведем список всех таблиц и сделаем выборку тех, которые не в кодировке utf8mb4_general_ci:

В качестве заметки привожу SQL запросы. Необходимо пройти по всем таблицам в базе и установить нужную кодировку.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

DESCRIBE merge_request_diff_commits;

ALTER DATABASE `gitlabhq_production` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

USE gitlabhq_production;

ALTER TABLE `gitlabhq_production`.`merge_request_diff_commits`

CHANGE COLUMN `committed_date` `committed_date` TIMESTAMP NULL DEFAULT NULL ;

ALTER TABLE `gitlabhq_production`.`merge_request_diff_commits` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

ALTER TABLE `gitlabhq_production`.`merge_request_diff_files` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

ALTER TABLE `gitlabhq_production`.`merge_request_diffs` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

ALTER TABLE `gitlabhq_production`.`merge_request_metrics` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

ALTER TABLE `gitlabhq_production`.`merge_requests` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

ALTER TABLE `gitlabhq_production`.`merge_requests_closing_issues` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

Иногда возникает ошибка вида:

Invalid default value for ‘updated_at’

или

Invalid default value for ‘created_at’

В данном случаи выполняем для проблемной таблицы запрос:

set sql_mode =»; ALTER TABLE approval_project_rules CHANGE COLUMN `created_at` `created_at` TIMESTAMP NULL DEFAULT NULL ;

set sql_mode =»; ALTER TABLE approval_project_rules CHANGE COLUMN `updated_at` `updated_at` TIMESTAMP NULL DEFAULT NULL ;

Также устанавливаем кодировку для всей базы gitlab:

ALTER DATABASE gitlabhq_production CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

Проверяем, что все ок

SELECT SCHEMA_NAME ‘gitlabhq_production’, default_character_set_name ‘charset’, DEFAULT_COLLATION_NAME ‘collation’ FROM information_schema.SCHEMATA;

+———————————+—————+——————————+

| gitlabhq_production | charset | collation          |

+———————————+—————+——————————+

| ...                 | ...     | ...                |

| gitlabhq_production | utf8mb4 | utf8mb4_general_ci |

| ...                 | ...     | ...                |

+———————————+—————+——————————+

Проверяем чтобы в настройках gitlab в конфиге также были указанные нужные нам кодировки:

cat config/database.yml

#

# PRODUCTION

#

production:

  adapter: mysql2

  encoding: utf8mb4

  collation: utf8mb4_general_ci

  reconnect: false

  database: gitlabhq_production

  host: mysql

  port: 3306

  username: username

  password: «Password»

  pool: 10

Источник

Summary

I have self-hosted gitlab, installed with apt. I don’t have git_data_dir in default location (see config). I can’t push docker image into docker registry, directory <shared_path>/registry does not exists. No migrations, no backup/restore procedure, last version.

Problem to solve

GitLab reject Docker image push from remote location with error 500. I’m not able to push image into my private docker registry. Have anyone idea why and how to solve it?

Pushing image from remote device

root@remote:cat Dockerfile
FROM alpine

root@remote:~/playground# docker login gitlab.mydomain.com:5050
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
root@remote:~/playground# docker build -t gitlab.mydomain.com:5050/testing/registry .
Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM alpine
 ---> e7d92cdc71fe
Successfully built e7d92cdc71fe
Successfully tagged gitlab.mydomain.com:5050/testing/registry:latest
root@remote:~/playground# docker push gitlab.mydomain.com:5050/testing/registry
The push refers to repository [gitlab.mydomain.com:5050/testing/registry]
5216338b40a7: Retrying in 1 second
received unexpected HTTP status: 500 Internal Server Error

Gitlab registry log

repo:/# tail /var/log/gitlab/registry/current
2020-01-21_13:46:16.49320 time="2020-01-21T14:46:16.493118369+01:00" level=warning msg="error authorizing context: authorization token required" go.version=go1.12.13 http.request.host="gitlab.mydomain.com:5050" http.request.id=fbe88f1e-ccf5-4fcd-8f3a-aa03d216388a http.request.method=GET http.request.remoteaddr=8.8.8.8 http.request.uri="/v2/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.19.0-6-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 (linux))"
2020-01-21_13:46:16.49351 127.0.0.1 - - [21/Jan/2020:14:46:16 +0100] "GET /v2/ HTTP/1.1" 401 87 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.19.0-6-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
2020-01-21_13:46:17.10631 time="2020-01-21T14:46:17.10627187+01:00" level=info msg="authorized request" go.version=go1.12.13 http.request.host="gitlab.mydomain.com:5050" http.request.id=7cc76f13-b5f3-4f4d-9309-d338b9c5c8b5 http.request.method=HEAD http.request.remoteaddr=8.8.8.8 http.request.uri="/v2/testing/registry/blobs/sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.19.0-6-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 (linux))" vars.digest="sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9" vars.name="testing/registry"
2020-01-21_13:46:17.10687 time="2020-01-21T14:46:17.106817596+01:00" level=error msg="response completed with error" auth.user.name=myname err.code=unknown err.detail="filesystem: open /mnt/data/git-data/gitlab-rails/shared/registry/docker/registry/v2/repositories/testing/registry/_layers/sha256/c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9/link: permission denied" err.message="unknown error" go.version=go1.12.13 http.request.host="gitlab.mydomain.com:5050" http.request.id=7cc76f13-b5f3-4f4d-9309-d338b9c5c8b5 http.request.method=HEAD http.request.remoteaddr=8.8.8.8 http.request.uri="/v2/testing/registry/blobs/sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.19.0-6-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 (linux))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.192904ms http.response.status=500 http.response.written=320 vars.digest="sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9" vars.name="testing/registry"
2020-01-21_13:46:17.10702 127.0.0.1 - - [21/Jan/2020:14:46:17 +0100] "HEAD /v2/testing/registry/blobs/sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9 HTTP/1.1" 500 320 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.19.0-6-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
2020-01-21_13:46:17.16482 time="2020-01-21T14:46:17.164783711+01:00" level=info msg="authorized request" go.version=go1.12.13 http.request.host="gitlab.mydomain.com:5050" http.request.id=e3e752c1-442a-46b1-b7c4-3f997e6e97a6 http.request.method=POST http.request.remoteaddr=8.8.8.8 http.request.uri="/v2/testing/registry/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.19.0-6-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 (linux))" vars.name="testing/registry"
2020-01-21_13:46:17.16537 time="2020-01-21T14:46:17.165324403+01:00" level=error msg="response completed with error" auth.user.name=myname err.code=unknown err.detail="filesystem: mkdir /mnt/data/git-data/gitlab-rails: permission denied" err.message="unknown error" go.version=go1.12.13 http.request.host="gitlab.mydomain.com:5050" http.request.id=e3e752c1-442a-46b1-b7c4-3f997e6e97a6 http.request.method=POST http.request.remoteaddr=8.8.8.8 http.request.uri="/v2/testing/registry/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.19.0-6-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 (linux))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.673484ms http.response.status=500 http.response.written=171 vars.name="testing/registry"
2020-01-21_13:46:17.16554 127.0.0.1 - - [21/Jan/2020:14:46:17 +0100] "POST /v2/testing/registry/blobs/uploads/ HTTP/1.1" 500 171 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.19.0-6-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"

Biggest problem (if I understand it right) is:

filesystem: open /mnt/data/git-data/gitlab-rails/shared/registry/docker/registry/v2/repositories/testing/registry/_layers/sha256/c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9/link: permission denied
filesystem: mkdir /mnt/data/git-data/gitlab-rails: permission denied

Directory content

There is no registry dir on path

repo:/# ll /mnt/data/git-data/gitlab-rails/shared/
total 40
drwxr-x--x  10 git  gitlab-www 4096 Jan 21 14:11 .
drwxr-xr-x   3 root root       4096 Sep 24  2018 ..
drwx------  11 git  root       4096 Dec 10 08:21 artifacts
drwx------   3 git  root       4096 Oct 24  2018 cache
drwx------   2 git  root       4096 Jul 30 10:36 dependency_proxy
drwx------   2 git  root       4096 Jul 30 10:36 external-diffs
drwx------ 259 git  root       4096 Oct 25  2018 lfs-objects
drwx------   2 git  root       4096 Dec  3  2018 packages
drwxr-x---   9 git  gitlab-www 4096 Dec 10 09:12 pages
drwx------   3 git  root       4096 Sep 24  2018 tmp

Gitlab config

root@repo:gitlab-ctl show-config

Starting Chef Client, version 14.13.11
resolving cookbooks for run list: ["gitlab::show_config"]
Synchronizing Cookbooks:
  - redis (0.1.0)
  - registry (0.1.0)
  - gitaly (0.1.0)
  - letsencrypt (0.1.0)
  - gitlab (0.0.1)
  - runit (4.3.0)
  - crond (0.1.0)
  - package (0.1.0)
  - postgresql (0.1.0)
  - consul (0.1.0)
  - nginx (0.1.0)
  - mattermost (0.1.0)
  - acme (4.0.0)
  - praefect (0.1.0)
  - monitoring (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...

{
  "gitlab": {
    "gitlab-shell": {
      "secret_token": "<some_hash>",
      "auth_file": "/var/opt/gitlab/.ssh/authorized_keys"
    },
    "gitlab-rails": {
      "lfs_enabled": true,
      "lfs_storage_path": "/mnt/data/git-data/gitlab-rails/shared/lfs-objects",
      "backup_path": "/mnt/data/gitlab-backup/",
      "backup_keep_time": 604800,
      "shared_path": "/mnt/data/git-data/gitlab-rails/shared",
      "secret_key_base": "<some_hash>",
      "db_key_base": "<some_hash>",
      "otp_key_base": "<some_hash>",
      "openid_connect_signing_key": "-----BEGIN RSA PRIVATE KEY-----n<some_hash>n-----END RSA PRIVATE KEY-----n",
      "gitlab_host": "gitlab.mydomain.com",
      "gitlab_email_from": "gitlab@gitlab.mydomain.com",
      "gitlab_https": true,
      "gitlab_port": 443,
      "artifacts_path": "/mnt/data/git-data/gitlab-rails/shared/artifacts",
      "external_diffs_storage_path": "/mnt/data/git-data/gitlab-rails/shared/external-diffs",
      "uploads_storage_path": "/opt/gitlab/embedded/service/gitlab-rails/public",
      "packages_storage_path": "/mnt/data/git-data/gitlab-rails/shared/packages",
      "dependency_proxy_storage_path": "/mnt/data/git-data/gitlab-rails/shared/dependency_proxy",
      "pages_path": "/mnt/data/git-data/gitlab-rails/shared/pages",
      "repositories_storages": {
        "default": {
          "path": "/mnt/data/git-data/repositories",
          "gitaly_address": "unix:/var/opt/gitlab/gitaly/gitaly.socket"
        }
      },
      "trusted_proxies": [

      ],
      "db_username": "gitlab",
      "db_host": null,
      "db_port": 5432
    },
    "gitlab-workhorse": {
      "secret_token": "<some_hash>",
      "auth_socket": "/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket"
    },
    "logging": {

    },
    "unicorn": {

    },
    "puma": {

    },
    "mailroom": {

    },
    "gitlab-pages": {
      "gitlab_secret": null,
      "gitlab_id": null,
      "auth_secret": "<some_hash>",
      "api_secret_key": "<some_hash>"
    },
    "external-url": "https://gitlab.mydomain.com",
    "registry-external-url": null,
    "mattermost-external-url": null,
    "pages-external-url": null,
    "runtime-dir": "/run",
    "git-data-dir": null,
    "bootstrap": {

    },
    "omnibus-gitconfig": {

    },
    "manage-accounts": {

    },
    "manage-storage-directories": {

    },
    "user": {
      "home": "/var/opt/gitlab",
      "git_user_email": "gitlab@gitlab.mydomain.com"
    },
    "gitlab-ci": {

    },
    "sidekiq": {

    },
    "mattermost-nginx": {
      "listen_port": null
    },
    "pages-nginx": {
      "listen_port": null
    },
    "registry-nginx": {

    },
    "remote-syslog": {

    },
    "logrotate": {

    },
    "high-availability": {

    },
    "web-server": {

    },
    "prometheus-monitoring": {

    },
    "pgbouncer": {

    },
    "pgbouncer-exporter": {

    },
    "storage-check": {
      "target": "unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket"
    },
    "nginx": {
      "redirect_http_to_https": true,
      "ssl_certificate": "/etc/gitlab/ssl/gitlab.mydomain.com.crt",
      "ssl_certificate_key": "/etc/gitlab/ssl/gitlab.mydomain.com.key",
      "proxy_set_headers": {
        "Host": "$http_host_with_default",
        "X-Real-IP": "$remote_addr",
        "X-Forwarded-For": "$proxy_add_x_forwarded_for",
        "Upgrade": "$http_upgrade",
        "Connection": "$connection_upgrade",
        "X-Forwarded-Proto": "https",
        "X-Forwarded-Ssl": "on"
      },
      "real_ip_trusted_addresses": [

      ],
      "listen_port": 443
    }
  },
  "roles": {
    "application": {

    },
    "redis-sentinel": {

    },
    "redis-master": {

    },
    "redis-slave": {

    },
    "geo-primary": {

    },
    "geo-secondary": {

    },
    "monitoring": {

    },
    "postgres": {

    },
    "pgbouncer": {

    },
    "consul": {

    }
  },
  "monitoring": {
    "prometheus": {
      "alertmanagers": [

      ],
      "flags": {
        "web.listen-address": "localhost:9090",
        "storage.tsdb.path": "/var/opt/gitlab/prometheus/data",
        "config.file": "/var/opt/gitlab/prometheus/prometheus.yml"
      }
    },
    "grafana": {
      "secret_key": "7dfc8ff446078cdabd489b77ec25fa37",
      "gitlab_secret": "<some_hash>",
      "gitlab_application_id": "<some_hash>",
      "admin_password": "<some_hash>",
      "metrics_basic_auth_password": null,
      "datasources": [
        {
          "name": "GitLab Omnibus",
          "type": "prometheus",
          "access": "proxy",
          "url": "http://localhost:9090",
          "isDefault": true
        }
      ]
    },
    "alertmanager": {
      "flags": {
        "web.listen-address": "localhost:9093",
        "storage.path": "/var/opt/gitlab/alertmanager/data",
        "config.file": "/var/opt/gitlab/alertmanager/alertmanager.yml"
      }
    },
    "node-exporter": {
      "flags": {
        "web.listen-address": "localhost:9100",
        "collector.mountstats": true,
        "collector.runit": true,
        "collector.runit.servicedir": "/opt/gitlab/sv",
        "collector.textfile.directory": "/var/opt/gitlab/node-exporter/textfile_collector"
      }
    },
    "redis-exporter": {
      "flags": {
        "web.listen-address": "localhost:9121",
        "redis.addr": "unix:///var/opt/gitlab/redis/redis.socket"
      }
    },
    "postgres-exporter": {
      "flags": {
        "web.listen-address": "localhost:9187",
        "extend.query-path": "/var/opt/gitlab/postgres-exporter/queries.yaml"
      }
    },
    "gitlab-exporter": {
      "probe_sidekiq": true
    },
    "gitlab-monitor": {

    }
  },
  "letsencrypt": {
    "auto_enabled": false,
    "enable": false
  },
  "package": {

  },
  "registry": {
    "health_storagedriver_enabled": false,
    "http_secret": "<some_hash>",
    "internal_certificate": "-----BEGIN CERTIFICATE-----<some_hash>n-----END CERTIFICATE-----n",
    "internal_key": "-----BEGIN RSA PRIVATE KEY-----n<some_hash>n-----END RSA PRIVATE KEY-----n"
  },
  "redis": {
    "rename_commands": {
      "KEYS": ""
    }
  },
  "postgresql": {
    "internal_certificate": "-----BEGIN CERTIFICATE-----n<some_hash>n-----END CERTIFICATE-----n",
    "internal_key": "-----BEGIN RSA PRIVATE KEY-----n<some_hash>n-----END RSA PRIVATE KEY-----n"
  },
  "repmgr": {

  },
  "repmgrd": {

  },
  "consul": {

  },
  "gitaly": {
    "storage": [
      {
        "name": "default",
        "path": "/mnt/data/git-data/repositories"
      }
    ]
  },
  "praefect": {

  },
  "crond": {

  },
  "mattermost": {
    "email_invite_salt": "<some_hash>",
    "file_public_link_salt": "<some_hash>",
    "sql_at_rest_encrypt_key": "<some_hash>",
    "sql_data_source": "user=gitlab_mattermost host=/var/opt/gitlab/postgresql port=5432 dbname=mattermost_production"
  }
}

Converging 0 resources

Running handlers:
Running handlers complete
Chef Client finished, 0/0 resources updated in 06 seconds

GitLab environment info

repo:/# gitlab-rake gitlab:env:info

System information
System:         Debian 8.11
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.6.3p62
Gem Version:    2.7.9
Bundler Version:1.17.3
Rake Version:   12.3.3
Redis Version:  3.2.12
Git Version:    2.24.1
Sidekiq Version:5.2.7
Go Version:     unknown

GitLab information
Version:        12.6.4-ee
Revision:       cc6b787e7b0
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     10.9
URL:            https://gitlab.mydomain.com
HTTP Clone URL: https://gitlab.mydomain.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.mydomain.com:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        10.3.0
Repository storage paths:
 - default:      /mnt/data/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Источник

I have been stuck for quite some time.
I wanted to use Gitlab for some personal projects and I am running it on my own server.

I just recently took interest in the CI/CD part of Gitlab and wanted to try it out.
However the runner keeps looping with a 500 internal server error.

WARNING: Checking for jobs... failed                runner=YmfuCQdb 
status=500 Internal Server Error
Feeding runners to channel                          builds=0
Dialing: tcp git.server.be:443 ...

I have googled this issue and cannot for the life of me solve it.

Some background information:
I migrated this server a while back, everything functioned just fine.
Any information you need I will try to provide.

Edit: I found the following in my logs [sanitized]

Started POST "/api/v4/jobs/request" for 0.0.0.0 at 2019-10-03 19:32:06 +0000

Processing by Gitlab::RequestForgeryProtection::Controller#index as JSON

Parameters: {"info"=>{"name"=>"gitlab-runner", "version"=>"12.3.0", "revision"=>"a8a019e0", "platform"=>"windows", "architecture"=>"amd64", "executor"=>"shell", "shell"=>"powershell", "features"=>{"variables"=>"[FILTERED]", "image"=>false, "services"=>false, "artifacts"=>true, "cache"=>true, "shared"=>true, "upload_multiple_artifacts"=>true, "upload_raw_artifacts"=>true, "session"=>false, "terminal"=>false, "refspecs"=>true, "masking"=>true, "proxy"=>false}}, "token"=>"[FILTERED]", "request_forgery_protection"=>{"info"=>{"name"=>"gitlab-runner", "version"=>"12.3.0", "revision"=>"a8a019e0", "platform"=>"windows", "architecture"=>"amd64", "executor"=>"shell", "shell"=>"powershell", "features"=>{"variables"=>"[FILTERED]", "image"=>false, "services"=>false, "artifacts"=>true, "cache"=>true, "shared"=>true, "upload_multiple_artifacts"=>true, "upload_raw_artifacts"=>true, "session"=>false, "terminal"=>false, "refspecs"=>true, "masking"=>true, "proxy"=>false}}, "token"=>"[FILTERED]"}}

Can't verify CSRF token authenticity.

This CSRF token verification failure is handled internally by `GitLab::RequestForgeryProtection`

Unlike the logs may suggest, this does not result in an actual 422 response to the user

For API requests, the only effect is that `current_user` will be `nil` for the duration of the request

Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms)

OpenSSL::Cipher::CipherError ():

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'

/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'

/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/base.rb:27:in `ensure_token'

/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:32:in `ensure_token'

/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable.rb:40:in `block in add_authentication_token_field'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:426:in `block in make_lambda'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:606:in `block (2 levels) in default_terminator'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:605:in `catch'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:605:in `block in default_terminator'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:199:in `block in halting'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:513:in `block in invoke_before'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:513:in `each'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:513:in `invoke_before'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:131:in `run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:816:in `_run_save_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/callbacks.rb:342:in `create_or_update'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/persistence.rb:275:in `save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/validations.rb:46:in `save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:310:in `block (2 levels) in save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:212:in `transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:310:in `block in save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:325:in `rollback_active_record_state!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:309:in `save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/suppressor.rb:44:in `save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:509:in `block in save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:528:in `block in around_save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:150:in `block in run_actions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in `catch_exceptions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in `run_actions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:133:in `run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:224:in `run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:128:in `block in run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:371:in `before'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:190:in `block in run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:292:in `block in pausable'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:292:in `catch'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:292:in `pausable'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:190:in `run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:127:in `run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:212:in `run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block (2 levels) in perform'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `catch'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block in perform'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:182:in `block in within_transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:171:in `block in within_transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:1868:in `block in within_transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:555:in `block in transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in `block in transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'

/opt/gitlab/embedded/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in `transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:212:in `transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:554:in `transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:1868:in `within_transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:170:in `within_transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:181:in `within_transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:62:in `perform'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:528:in `around_save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:509:in `save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:154:in `block (2 levels) in run_actions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:154:in `each'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:154:in `block in run_actions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in `catch_exceptions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in `run_actions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:60:in `perform'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:163:in `perform'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:155:in `fire'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:219:in `block in add_actions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:729:in `block (2 levels) in define_helper'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:224:in `block in add_actions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:729:in `block (2 levels) in define_helper'

/opt/gitlab/embedded/service/gitlab-rails/app/services/ci/register_job_service.rb:95:in `assign_runner!'

/opt/gitlab/embedded/service/gitlab-rails/app/services/ci/register_job_service.rb:50:in `block in execute'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/delegation.rb:71:in `each'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/delegation.rb:71:in `each'

/opt/gitlab/embedded/service/gitlab-rails/app/services/ci/register_job_service.rb:44:in `execute'

/opt/gitlab/embedded/service/gitlab-rails/lib/api/runner.rb:120:in `block (2 levels) in <class:Runner>'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `block (2 levels) in generate_api_method'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:170:in `instrument'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:56:in `block in generate_api_method'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:262:in `block in run'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:170:in `instrument'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:243:in `run'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:313:in `block in build_stack'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/resource.rb:20:in `_call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/resource/bearer.rb:8:in `_call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/abstract/handler.rb:17:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:38:in `block in call!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `catch'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `call!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `catch'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `call!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:227:in `call!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:221:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router/route.rb:72:in `exec'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:121:in `process_route'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:74:in `block in identity'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:93:in `transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:72:in `identity'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:57:in `block in call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:137:in `with_optimization'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:56:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:119:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:45:in `call!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:40:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/mapper.rb:48:in `serve'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:52:in `block in serve'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `each'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `serve'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:840:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/transaction.rb:57:in `run'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:17:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:103:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:16:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:20:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:13:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/correlation_id.rb:16:in `block in call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.3.0/lib/labkit/correlation/correlation_id.rb:18:in `use_id'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/correlation_id.rb:15:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/batch-loader-1.4.0/lib/batch_loader/middleware.rb:11:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/apollo_upload_server-2.0.0.beta.3/lib/apollo_upload_server/middleware.rb:20:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-attack-4.4.1/lib/rack/attack.rb:107:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `catch'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-cors-1.0.2/lib/rack/cors.rb:97:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/tempfile_reaper.rb:15:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/etag.rb:25:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/conditional_get.rb:38:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/http/content_security_policy.rb:18:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:40:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:18:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:232:in `context'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:226:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/cookies.rb:670:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:98:in `run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/basic_health_check.rb:25:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:38:in `call_app'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:26:in `block in call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `block in tagged'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:28:in `tagged'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `tagged'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:26:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_context.rb:26:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/request_id.rb:27:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/runtime.rb:22:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/executor.rb:14:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sentry-raven-2.9.0/lib/raven/integrations/rack.rb:51:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:524:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `public_send'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `method_missing'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:12:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:606:in `process_client'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:701:in `worker_loop'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:142:in `start'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/bin/unicorn:126:in `<top (required)>'

/opt/gitlab/embedded/bin/unicorn:23:in `load'

/opt/gitlab/embedded/bin/unicorn:23:in `<main>'

Источник

This topic has been deleted. Only users with topic management privileges can see it.

  • Somehow my Gitlab seems to be broken. When I try to access the Admin page for the runners I get a 500 error. While trying to fix it I figured out that my Gitlab does not start when I try to enable the recovery mode. Any way to fix this?

  • @rav Is there any error in the logs when you get a 500? If you disable recovery mode, does the app come back up? It is expected that the app won’t start in recovery mode. You have to go the web terminal and then run /app/pkg/start.sh to start the app.

    I guess the page is /admin/runners and not /admin/runner ?

  • @girish The error in the log seems to be not so usefull.

    Jan 23 10:55:32 172.18.0.1 — — [23/Jan/2021:09:55:32 +0000] «GET /admin/runners HTTP/1.1» 500 2926 «https://git.aaa.bb/admin» «Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15
    Jan 23 10:55:35 172.18.0.1 — — [23/Jan/2021:09:55:35 +0000] «POST /api/v4/jobs/request HTTP/1.1» 500 39 «-» «gitlab-runner 13.7.0 (13-7-stable; go1.13.8; linux/amd64)»

    In recovery mode in the webterminal I am unable to type or see anything this is what seems strange. Yes it is /admin/runners sorry.

  • @rav GitLab logs are under /run/gitlab/log/. There are two files there application.log and production.log. Maybe you can load the page and then look for the backtrace in those files.

  • @girish found this in production.log

    Started GET "/admin/runners" for x.x.x.x at 2021-01-26 10:41:57 +0000
Processing by Admin::RunnersController#index as HTML
Completed 500 Internal Server Error in 88ms (ActiveRecord: 1.9ms | Elasticsearch: 0.0ms | Allocations: 48002)
  
ActionView::Template::Error ():
    36: 
    37:   .col-sm-6
    38:     .bs-callout
    39:       = render partial: 'ci/runner/how_to_setup_runner',
    40:                 locals: { registration_token: Gitlab::CurrentSettings.runners_registration_token,
    41:                           type: 'shared',
    42:                           reset_token_url: reset_registration_token_admin_application_settings_path }
  
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:38:in `ensure_token!'
app/models/concerns/token_authenticatable.rb:48:in `block in add_authentication_token_field'
app/models/application_setting_implementation.rb:334:in `runners_registration_token'
lib/gitlab/current_settings.rb:20:in `method_missing'
app/views/admin/runners/index.html.haml:39
app/controllers/application_controller.rb:133:in `render'
app/controllers/application_controller.rb:494:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:485:in `set_session_storage'
lib/gitlab/i18n.rb:73:in `with_locale'
lib/gitlab/i18n.rb:79:in `with_user_locale'
app/controllers/application_controller.rb:479:in `set_locale'
lib/gitlab/error_tracking.rb:52:in `with_context'
app/controllers/application_controller.rb:544:in `sentry_context'
app/controllers/application_controller.rb:472:in `block in set_current_context'
lib/gitlab/application_context.rb:54:in `block in use'
lib/gitlab/application_context.rb:54:in `use'
lib/gitlab/application_context.rb:21:in `with_context'
app/controllers/application_controller.rb:464:in `set_current_context'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:234:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

  • @rav I found the exact same error in https://stackoverflow.com/questions/54216933/internal-server-error-500-while-accessing-gitlab-admin-runners .

    As per the instruction in the stackoverflow post, can you try this in Cloudron’s Web terminal?:

    # su - git  
$ cd gitlab
$ bundle exec rails c -e production # (takes about 10 sec to bring up rails terminal)
irb(main):001:0> ApplicationSetting.current.reset_runners_registration_token!

  • @girish said in Gitlab /admin/runner error 500:

    ApplicationSetting.current.reset_runners_registration_token!

    Tried this and the Rails console said => True.
    The /admin/runners page is still not available. Any other logs I could check?

  • @rav said in Gitlab /admin/runner error 500:

    The /admin/runners page is still not available. Any other logs I could check?

    So, same backtrace? Have you tried restarting the app just to be sure?

  • @girish looks the same to me yes. did a restart of the app but it did not help.

    Started GET "/admin/runners" for a.b.c.d at 2021-01-30 10:47:26 +0000
Processing by Admin::RunnersController#index as HTML
Completed 500 Internal Server Error in 124ms (ActiveRecord: 29.1ms | Elasticsearch: 0.0ms | Allocations: 96262)
  
ActionView::Template::Error ():
    16:   .table-section.section-10
    17:     .table-mobile-header{ role: 'rowheader' }= _('Runner token')
    18:     .table-mobile-content
    19:       = link_to runner.short_sha, admin_runner_path(runner)
    20: 
    21:   .table-section.section-20
    22:     .table-mobile-header{ role: 'rowheader' }= _('Description')
  
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable.rb:35:in `block in add_authentication_token_field'
app/models/ci/runner.rb:265:in `short_sha'
app/views/admin/runners/_runner.html.haml:19
app/views/admin/runners/index.html.haml:134
app/views/admin/runners/index.html.haml:133
app/controllers/application_controller.rb:125:in `render'
app/controllers/application_controller.rb:482:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:473:in `set_session_storage'
lib/gitlab/i18n.rb:73:in `with_locale'
lib/gitlab/i18n.rb:79:in `with_user_locale'
app/controllers/application_controller.rb:467:in `set_locale'
lib/gitlab/error_tracking.rb:52:in `with_context'
app/controllers/application_controller.rb:532:in `sentry_context'
app/controllers/application_controller.rb:460:in `block in set_current_context'
lib/gitlab/application_context.rb:56:in `block in use'
lib/gitlab/application_context.rb:56:in `use'
lib/gitlab/application_context.rb:22:in `with_context'
app/controllers/application_controller.rb:451:in `set_current_context'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:172:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

  • @rav I can’t make out what the issue is. Do you think you can reach out to us at support@cloudron.io and give us SSH access so I can debug gitlab a bit to see what the issue is? Alternately, you have to ask on the GitLab forums on what to do for this.

  • @girish okay opened the support ticket from my cloudron

    • Источник

    Понравилась статья? Поделить с друзьями:

    Читайте также:

  • Github ошибка при скачивании
  • Github ошибка 403
  • Github как изменить язык на сайте
  • Github как изменить название pull request
  • Github как изменить имя пользователя

    • 0 0 голоса
    Рейтинг статьи
    Подписаться
    Уведомить о
    guest

    0 комментариев
    Старые
    Новые Популярные
    Межтекстовые Отзывы
    Посмотреть все комментарии