Gitlab ci cd 500 error after migrate - Исправление ошибок и поиск оптимальных решений проблем

Hi every one i migrated gitlab-ce into a new instance with new domain name

Migration :

Gitlab Backup —> Restore

Gitlab Version:

13.6.2

Problem :

when Clicking a Project it returns :

"500 Whoops, something went wrong on our end "

Gitlab status

run: alertmanager: (pid 1766) 3439s; run: log: (pid 1764) 3439s
run: crond: (pid 1775) 3439s; run: log: (pid 1770) 3439s
run: gitaly: (pid 1724) 3439s; run: log: (pid 1723) 3439s
run: gitlab-exporter: (pid 1750) 3439s; run: log: (pid 1749) 3439s
run: gitlab-workhorse: (pid 1761) 3439s; run: log: (pid 1748) 3439s
run: grafana: (pid 1767) 3439s; run: log: (pid 1765) 3439s
run: logrotate: (pid 1752) 3439s; run: log: (pid 1751) 3439s
run: nginx: (pid 1746) 3439s; run: log: (pid 1745) 3439s
run: node-exporter: (pid 1742) 3439s; run: log: (pid 1740) 3439s
run: postgres-exporter: (pid 1769) 3439s; run: log: (pid 1768) 3439s
run: postgresql: (pid 1726) 3439s; run: log: (pid 1720) 3439s
run: prometheus: (pid 1774) 3439s; run: log: (pid 1773) 3439s
run: puma: (pid 1747) 3439s; run: log: (pid 1734) 3439s
run: redis: (pid 1718) 3439s; run: log: (pid 1716) 3439s
run: redis-exporter: (pid 1757) 3439s; run: log: (pid 1756) 3439s
run: registry: (pid 1760) 3439s; run: log: (pid 1759) 3439s
run: sidekiq: (pid 1741) 3439s; run: log: (pid 1731) 3439s

Application Check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.13.0 ? ... OK (13.13.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... Server: ldapmain
Exception: getaddrinfo: Name or service not known

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
Mahdi Hannoun / test ... yes
octopeek-xenonstack / bdaas-application / frontend ... yes
octopeek-xenonstack / bdaas-application / backend ... yes
Julien Hay / ProxyBench ... yes
flowbird-bi / data-management ... yes
flowbird-bi / reports-management ... yes
smart-data / abbakan ... yes
Hamid Zemirline / deployment-automation ... yes
rbc / ged-ratios ... yes
Oumar Fall / bytel ... yes
bouygues_dataviz / frontend ... yes
bouygues_dataviz / backend ... yes
E-TIPI / etipi-react-native-application ... yes
Julien Hay / VilleFR ... yes
flowbird-bi / report-sources ... yes
flowbird-bi / report-queries ... yes
octopeek-io / front-end ... yes
octopeek-io / old-back-end ... yes
showroomprive / archive / frontend-lot1 ... yes
Laure Normand / FormationDB ... yes
librestamp / librestamp ... yes
flowbird-bi / deployment-automation ... yes
showroomprive / archive / parse-prepare-process ... yes
showroomprive / scraping-vente-privee ... yes
octopeek-io / site ... yes
Julien Hay / TwitterCrawler ... yes
showroomprive / front-end ... yes
Sedki Boubakri / srp ... yes
showroomprive / back-end ... yes
showroomprive / extract-image-srp ... yes
showroomprive / archive / application-libs-broken ... yes
Guillaume Ambrois / my-test-project ... yes
flowbird-bi / data-processing ... yes
Hamid Zemirline / src_flb_bi ... yes
Julien Hay / thang-matching ... yes
Pierre Gauthier / scraping_linkedin ... yes
smaps / back-end ... yes
flowbird-uk / poc ... yes
matching-rh / archive / front-end ... yes
matching-rh / archive / back-end ... yes
matching-rh / archive / application-libs ... yes
showroomprive / functional-tests ... yes
Guillaume Ambrois / application-libs ... yes
E-TIPI / application-libs ... yes
showroomprive / data-ingestion ... yes
Guillaume Ambrois / back-end ... yes
librestamp / application-libs ... yes
Guillaume Ambrois / application-libs-DO_NOT_TOUCH ... yes
showroomprive / application-libs ... yes
Pierre Gauthier / Matching_linkedin_page_jaune ... yes
matching-rh / archive / front-end-demo ... yes
infra / ansible-hortonworks ... yes
misc / requests ... yes
Krystian Luczyszyn / scrapping ... yes
Pierre Gauthier / MAJ_Index_Linkedin ... yes
Guillaume Ambrois / private ... yes
test-gitlab / test ... yes
Djamal KESSOUM / Infra_freeipa-ansible ... yes
Guillaume Ambrois / front-end ... yes
showroomprive / scraping-privalia ... yes
side-effects-detection / application_libs ... yes
side-effects-detection / backend ... yes
side-effects-detection / frontend ... yes
octopeek-io / old-front-end ... yes
Besma Khalfi / data-ingestion ... yes
showroomprive / scheduled-scripts ... yes
Moncef Mouffok / emailing ... yes
infra / octopeek-dev-sandbox ... yes
infra / devices-conf ... yes
corp-agency / data-acquisition ... yes
Guillaume Ambrois / gsuite-to-sheets ... yes
infra / ansible-octopeek-infra ... yes
corp-agency / data-normalization ... yes
infra / tools ... yes
showroomprive / data-restitution ... yes
Julien Hay / LinkedInCrawler ... yes
Oumar Fall / octopeek-commons ... yes
corp-agency / misc-scripts ... yes
octopeek-commons / front-octopeek-menu ... yes
octopeek-commons / front-octopeek-multi-language ... yes
octopeek-commons / front-file-drop-upload ... yes
octopeek-templates / octopeek-workspace ... yes
octopeek-io / deep-end ... yes
Oumar Fall / node-backend ... yes
corp-agency / data-processing ... yes
octopeek-io / back-end ... yes
Bertrand Laporte / cicd-test ... yes
Besma Khalfi / BesmaProject ... yes
Besma Khalfi / CI-CD testing  ... yes
these_mmouffok / memoire ... yes
data-acquisition / sirene ... yes
data-acquisition / pages-jaunes ... yes
Mahdi MENACER / workshop-git ... yes
matching-rh / archive / deep-end ... yes
these_mmouffok / deep_domain_adaptation_code ... yes
data-acquisition-tools / email-creation ... yes
RH-Matching-demo / front-end ... yes
these_mmouffok / sketch_based_3D_shape_code ... yes
RH-Matching-demo / back-end ... yes
RH-Matching-demo / Deep-end-demo ... yes
data-ingestion / company ... yes
academai / jupyter ... yes
matching-rh / deep-end-emploi-collectivites ... yes
Carole Lambert / deep-end-emploi-collectivites ... yes
Julien Hay / Twirona ... yes
matching-rh / back-end-demo-emploi-collectivites ... yes
matching-rh / front-end-demo-emploi-collectivites ... yes
RH-Matching-demo / front-end-demo-emploi-collectivites ... yes
octopeek-io / catalogue-services ... yes
data-performance / AVY-Salon Aveyron ... yes
octopeek-io / Mongodb_catalogue ... yes
octopeek-io / catalogue-services-front-admin ... yes
octopeek-io / api_catalogue_services ... yes
Oumar Fall / demo-cicd ... yes
octopeek-web / apache-configuration ... yes
infra / octopeek-template ... yes
infra / octopeek-provisioning ... yes
Oumar Fall / api_ioss ... yes
octopeek-xenonstack / bdaas-infra-deploy / ambari-deploy ... yes
octopeek-xenonstack / bdaas-infra-deploy / kubespray ... yes
octopeek-xenonstack / bdaas-infra-deploy / helm-charts ... yes
octopeek-xenonstack / bdaas-infra-deploy / k8s-storage ... yes
octopeek-xenonstack / bdaas-app-deploy / frontend-helm ... yes
octopeek-xenonstack / bdaas-app-deploy / backend-helm ... yes
Samy Tafasca / AutoEDA ... yes
Oumar Fall / demo-ansible-gitlab-ci ... yes
octopeek-xenonstack / bdaas-infra-deploy / ci-experiment ... yes
Hamid Zemirline / aaa-service ... yes
IOSS / ioss-ui-socle ... yes
IOSS / ioss-user-mgt-service ... yes
IOSS / ioss-authentication ... yes
Erwan Benkara / AutoEDA ... yes
octopeek-xenonstack / bdaas-application / component-deploy-backend ... yes
octopeek-xenonstack / bdaas-app-deploy / component-deploy-backend-helm ... yes
Hamid Zemirline / helm-repo ... yes
IOSS / ioss-user-mgt-api ... yes
IOSS / ioss-deploy ... yes
IOSS / ioss-user-log-service ... yes
IOSS / ioss-dev-stack ... yes
IOSS / helm-charts ... yes
SedkiBoubakri / EdaPerformanceTest ... yes
weather-data-acquisition / Web scraping ... yes
SedkiBoubakri / Profiling ... yes
IOSS / cdap-tests ... yes
Thomas Levy / explainability ... yes
IOSS / cdap-operator ... yes
IOSS / cdap-build ... yes
IOSS / cdap ... yes
IOSS / cdap-plugins ... yes
CDAP-plugins / product-auto-extract ... yes
CDAP-plugins / elasticsearch-source-plugin ... yes
CDAP-plugins / diffbotcrawl-action ... yes
CDAP-plugins / octopeek-http-plugins ... yes
CDAP-plugins / explainability-poc ... yes
IOSS / pipelines ... yes
CDAP-plugins / webdataextractor ... yes
Julien Hay / POCImmo ... yes
Samy Tafasca / autoeda_spark ... yes
Stephane Durfort / Python Plugins POC ... yes
CDAP-plugins / python-plugins ... yes
IOSS / cdap-plugin-test ... yes
CDAP-plugins / interpreted-Python-for-matching ... yes
IOSS / hub-templates ... yes
misc / ohb-cluster-archives ... yes
CDAP-plugins / diffbot-tools ... yes
IOSS / devops-scripts ... yes
IOSS / mvn-depgraph ... yes
CDAP-plugins / product-classification ... yes
Ahmed Zouari / Product_Classification ... yes
CDAP-plugins / scraping-confidence ... yes
CDAP-plugins / text-normalization-directive ... yes
CDAP-plugins / random-hash ... yes
IOSS / cdap_cookbook ... yes
CDAP-plugins / text-normalization-directive_with_parameter ... yes
CDAP-plugins / sentiment-analysis ... yes
CDAP-plugins / oit-commons ... yes
IOSS / french-product-classification ... yes
CDAP-plugins / TextNormalizationPlugin ... yes
CDAP-plugins / ai-enrich-plugin ... yes
CDAP-plugins / aienrich ... yes
IOSS / automapping ... yes
Stephane Durfort / Mvn Build Number Test ... yes
CDAP-plugins / widgets-demo ... yes
CDAP-plugins / data-generator ... yes
Lorenzo Faccioli / hubspot_migration ... yes
CDAP-plugins / java-basics ... yes
cdapio_old / cdap-build ... yes
IOSS / cdap-mirroring ... yes
cdapio_old / hydrator-plugins ... yes
cdapio_old / cdap-security-extn ... yes
cdapio_old / wrangler ... yes
cdapio_old / bigquery-delta-plugins ... yes
cdapio_old / delta ... yes
cdapio_old / database-delta-plugins ... yes
cdapio_old / amazon-s3-plugins ... yes
cdapio_old / azure ... yes
cdapio_old / condition-plugins ... yes
cdapio_old / google-cloud ... yes
cdapio_old / kafka-plugins ... yes
cdapio_old / mmds ... yes
Lorenzo Faccioli / Get refresh token ... yes
cdapio_old / cdap ... yes
cdapio_old / cdap-ui ... yes
CDAP-plugins / Get Refresh Token ... yes
CDAP-plugins / Google Drive refresh token ... yes
Lorenzo Faccioli / test ... yes
test-group / Repotest ... yes
cdapio_old / mirroring-test ... yes
cdapio / cdap-build ... yes
cdapio / cdap ... yes
cdapio / cdap-ui ... yes
cdapio / hydrator-plugins ... yes
cdapio / cdap-security-extn ... yes
cdapio / wrangler ... yes
cdapio / bigquery-delta-plugins ... yes
cdapio / delta ... yes
cdapio / database-delta-plugins ... yes
cdapio / amazon-s3-plugins ... yes
cdapio / azure ... yes
cdapio / condition-plugins ... yes
cdapio / google-cloud ... yes
cdapio / kafka-plugins ... yes
cdapio / mmds ... yes
Ossama Assaghir / mirroring-test ... yes
CDAP-plugins / elasticsearch-plugins-release-1.10 ... yes
CDAP-plugins / test_besma ... yes
CDAP-plugins / elasticsearch-plugins-release-1.10.0 ... yes
cdapio / elasticsearch-plugins ... yes
IOSS / SpecializedScraperPOC ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.7.2)
Git version >= 2.29.0 ? ... yes (2.29.0)
Git user has default SSH configuration? ... yes
Active users: ... 40
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

For ldap error it’s just because ldap server is shut down at the moment

here is gitlab-reails/production.log when i click a project

==> /var/log/gitlab/gitlab-rails/production.log <==

ActionView::Template::Error ():
    128:           = render_if_exists 'layouts/nav/sidebar/project_jira_issues_link', issue_tracker: issue_tracker
    129:         - else
    130:           = nav_link do
    131:             = link_to issue_tracker.issue_tracker_path, target: '_blank', rel: 'noopener noreferrer', class: 'shortcuts-external_tracker' do
    132:               .nav-icon-container
    133:                 = sprite_icon('external-link')
    134:               %span.nav-item-name

app/models/project_services/data_fields.rb:14:in `url'
app/models/project_services/issue_tracker_service.rb:63:in `issue_tracker_path'
app/views/layouts/nav/sidebar/_project.html.haml:131
app/helpers/tab_helper.rb:78:in `nav_link'
app/views/layouts/nav/sidebar/_project.html.haml:130
app/views/layouts/_page.html.haml:3
app/views/layouts/application.html.haml:13
app/views/layouts/project.html.haml:16
app/controllers/application_controller.rb:134:in `render'
app/controllers/application_controller.rb:554:in `block in allow_gitaly_ref_name_caching'
lib/gitlab/gitaly_client.rb:318:in `allow_ref_name_caching'
app/controllers/application_controller.rb:553:in `allow_gitaly_ref_name_caching'
app/controllers/application_controller.rb:499:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:490:in `set_session_storage'
lib/gitlab/i18n.rb:73:in `with_locale'
lib/gitlab/i18n.rb:79:in `with_user_locale'
app/controllers/application_controller.rb:484:in `set_locale'
lib/gitlab/error_tracking.rb:52:in `with_context'
app/controllers/application_controller.rb:549:in `sentry_context'
app/controllers/application_controller.rb:477:in `block in set_current_context'
lib/gitlab/application_context.rb:54:in `block in use'
lib/gitlab/application_context.rb:54:in `use'
lib/gitlab/application_context.rb:21:in `with_context'
app/controllers/application_controller.rb:469:in `set_current_context'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:234:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

thank you in advance !

Источник

Troubleshooting CI/CD (FREE)

GitLab provides several tools to help make troubleshooting your pipelines easier.

This guide also lists common issues and possible solutions.

Verify syntax

An early source of problems can be incorrect syntax. The pipeline shows a yaml invalid
badge and does not start running if any syntax or formatting problems are found.

Edit `.gitlab-ci.yml` with the pipeline editor

The pipeline editor is the recommended editing
experience (rather than the single file editor or the Web IDE). It includes:

Code completion suggestions that ensure you are only using accepted keywords.
Automatic syntax highlighting and validation.
The CI/CD configuration visualization,
a graphical representation of your .gitlab-ci.yml file.

Edit `.gitlab-ci.yml` locally

If you prefer to edit your pipeline configuration locally, you can use the
GitLab CI/CD schema in your editor to verify basic syntax issues. Any
editor with Schemastore support uses
the GitLab CI/CD schema by default.

If you need to link to the schema directly, it
is at:

https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/editor/schema/ci.json.

To see the full list of custom tags covered by the CI/CD schema, check the
latest version of the schema.

Verify syntax with CI Lint tool

The CI Lint tool is a simple way to ensure the syntax of a CI/CD configuration
file is correct. Paste in full .gitlab-ci.yml files or individual jobs configuration,
to verify the basic syntax.

When a .gitlab-ci.yml file is present in a project, you can also use the CI Lint
tool to simulate the creation of a full pipeline.
It does deeper verification of the configuration syntax.

Verify variables

A key part of troubleshooting CI/CD is to verify which variables are present in a
pipeline, and what their values are. A lot of pipeline configuration is dependent
on variables, and verifying them is one of the fastest ways to find the source of
a problem.

Export the full list of variables
available in each problematic job. Check if the variables you expect are present,
and check if their values are what you expect.

GitLab CI/CD documentation

The complete .gitlab-ci.yml reference contains a full list of
every keyword you can use to configure your pipelines.

You can also look at a large number of pipeline configuration examples
and templates.

Documentation for pipeline types

Some pipeline types have their own detailed usage guides that you should read
if you are using that type:

Multi-project pipelines: Have your pipeline trigger
a pipeline in a different project.
Parent/child pipelines: Have your main pipeline trigger
and run separate pipelines in the same project. You can also
dynamically generate the child pipeline’s configuration
at runtime.
Merge request pipelines: Run a pipeline
in the context of a merge request.
- Merged results pipelines:
  Merge request pipelines that run on the combined source and target branch
- Merge trains:
  Multiple merged results pipelines that queue and run automatically before
  changes are merged.

Troubleshooting Guides for CI/CD features

Troubleshooting guides are available for some CI/CD features and related topics:

Container Registry
GitLab Runner
Merge Trains
Docker Build
Environments

Common CI/CD issues

A lot of common pipeline issues can be fixed by analyzing the behavior of the rules
or only/except configuration. You shouldn’t use these two configurations in the same
pipeline, as they behave differently. It’s hard to predict how a pipeline runs with
this mixed behavior.

If your rules or only/except configuration makes use of predefined variables
like CI_PIPELINE_SOURCE, CI_MERGE_REQUEST_ID, you should verify them
as the first troubleshooting step.

Jobs or pipelines don’t run when expected

The rules or only/except keywords are what determine whether or not a job is
added to a pipeline. If a pipeline runs, but a job is not added to the pipeline,
it’s usually due to rules or only/except configuration issues.

If a pipeline does not seem to run at all, with no error message, it may also be
due to rules or only/except configuration, or the workflow: rules keyword.

If you are converting from only/except to the rules keyword, you should check
the rules configuration details carefully. The behavior
of only/except and rules is different and can cause unexpected behavior when migrating
between the two.

The common if clauses for rules
can be very helpful for examples of how to write rules that behave the way you expect.

Two pipelines run at the same time

Two pipelines can run when pushing a commit to a branch that has an open merge request
associated with it. Usually one pipeline is a merge request pipeline, and the other
is a branch pipeline.

This situation is usually caused by the rules configuration, and there are several ways to
prevent duplicate pipelines.

A job is not in the pipeline

GitLab determines if a job is added to a pipeline based on the only/except
or rules defined for the job. If it didn’t run, it’s probably
not evaluating as you expect.

No pipeline or the wrong type of pipeline runs

Before a pipeline can run, GitLab evaluates all the jobs in the configuration and tries
to add them to all available pipeline types. A pipeline does not run if no jobs are added
to it at the end of the evaluation.

If a pipeline did not run, it’s likely that all the jobs had rules or only/except that
blocked them from being added to the pipeline.

If the wrong pipeline type ran, then the rules or only/except configuration should
be checked to make sure the jobs are added to the correct pipeline type. For
example, if a merge request pipeline did not run, the jobs may have been added to
a branch pipeline instead.

It’s also possible that your workflow: rules configuration
blocked the pipeline, or allowed the wrong pipeline type.

Pipeline with many jobs fails to start

A Pipeline that has more jobs than the instance’s defined CI/CD limits
fails to start.

To reduce the number of jobs in your pipeline, you can split your .gitlab-ci.yml
configuration using parent-child pipelines.

A job runs unexpectedly

A common reason a job is added to a pipeline unexpectedly is because the changes
keyword always evaluates to true in certain cases. For example, changes is always
true in certain pipeline types, including scheduled pipelines and pipelines for tags.

The changes keyword is used in combination with only/except
or rules). It’s recommended to use changes with
rules or only/except configuration that ensures the job is only added to branch
pipelines or merge request pipelines.

«fatal: reference is not a tree» error

Introduced in GitLab 12.4.

Previously, you’d have encountered unexpected pipeline failures when you force-pushed
a branch to its remote repository. To illustrate the problem, suppose you’ve had the current workflow:

A user creates a feature branch named example and pushes it to a remote repository.
A new pipeline starts running on the example branch.
A user rebases the example branch on the latest default branch and force-pushes it to its remote repository.
A new pipeline starts running on the example branch again, however,
the previous pipeline (2) fails because of fatal: reference is not a tree: error.

This occurs because the previous pipeline cannot find a checkout-SHA (which is associated with the pipeline record)
from the example branch that the commit history has already been overwritten by the force-push.
Similarly, Merged results pipelines
might have failed intermittently due to the same reason.

As of GitLab 12.4, we’ve improved this behavior by persisting pipeline refs exclusively.
To illustrate its life cycle:

A pipeline is created on a feature branch named example.
A persistent pipeline ref is created at refs/pipelines/<pipeline-id>,
which retains the checkout-SHA of the associated pipeline record.
This persistent ref stays intact during the pipeline execution,
even if the commit history of the example branch has been overwritten by force-push.
The runner fetches the persistent pipeline ref and gets source code from the checkout-SHA.
When the pipeline finishes, its persistent ref is cleaned up in a background process.

Merge request pipeline messages

The merge request pipeline widget shows information about the pipeline status in
a merge request. It’s displayed above the ability to merge status widget.

«Checking ability to merge automatically» message

There is a known issue
where a merge request can be stuck with the Checking ability to merge automatically
message.

If your merge request has this message and it does not disappear after a few minutes,
you can try one of these workarounds:

Refresh the merge request page.
Close & Re-open the merge request.
Rebase the merge request with the /rebase quick action.
If you have already confirmed the merge request is ready to be merged, you can merge
it with the /merge quick action.

«Checking pipeline status» message

This message is shown when the merge request has no pipeline associated with the
latest commit yet. This might be because:

GitLab hasn’t finished creating the pipeline yet.
You are using an external CI service and GitLab hasn’t heard back from the service yet.
You are not using CI/CD pipelines in your project.
You are using CI/CD pipelines in your project, but your configuration prevented a pipeline from running on the source branch for your merge request.
The latest pipeline was deleted (this is a known issue).
The source branch of the merge request is on a private fork.

After the pipeline is created, the message updates with the pipeline status.

Merge request status messages

The merge request status widget shows the Merge button and whether or not a merge
request is ready to merge. If the merge request can’t be merged, the reason for this
is displayed.

If the pipeline is still running, Merge is replaced with the
Merge when pipeline succeeds button.

If Merge Trains
are enabled, the button is either Add to merge train or Add to merge train when pipeline succeeds. (PREMIUM)

«A CI/CD pipeline must run and be successful before merge» message

This message is shown if the Pipelines must succeed
setting is enabled in the project and a pipeline has not yet run successfully.
This also applies if the pipeline has not been created yet, or if you are waiting
for an external CI service. If you don’t use pipelines for your project, then you
should disable Pipelines must succeed so you can accept merge requests.

«Merge blocked: pipeline must succeed. Push a new commit that fixes the failure» message

This message is shown if the merge request pipeline,
merged results pipeline,
or merge train pipeline
has failed or been canceled.

If a merge request pipeline or merged result pipeline was canceled or failed, you can:

Re-run the entire pipeline by selecting Run pipeline in the pipeline tab in the merge request.
Retry only the jobs that failed. If you re-run the entire pipeline, this is not necessary.
Push a new commit to fix the failure.

If the merge train pipeline has failed, you can:

Check the failure and determine if you can use the /merge quick action to immediately add the merge request to the train again.
Re-run the entire pipeline by selecting Run pipeline in the pipeline tab in the merge request, then add the merge request to the train again.
Push a commit to fix the failure, then add the merge request to the train again.

If the merge train pipeline was canceled before the merge request was merged, without a failure, you can:

Add it to the train again.

Project `group/project` not found or access denied

This message is shown if configuration is added with include and one of the following:

The configuration refers to a project that can’t be found.
The user that is running the pipeline is unable to access any included projects.

To resolve this, check that:

The path of the project is in the format my-group/my-project and does not include
any folders in the repository.
The user running the pipeline is a member of the projects
that contain the included files. Users must also have the permission
to run CI/CD jobs in the same projects.

«The parsed YAML is too big» message

This message displays when the YAML configuration is too large or nested too deeply.
YAML files with a large number of includes, and thousands of lines overall, are
more likely to hit this memory limit. For example, a YAML file that is 200kb is
likely to hit the default memory limit.

To reduce the configuration size, you can:

Check the length of the expanded CI/CD configuration in the pipeline editor’s
merged YAML tab. Look for
duplicated configuration that can be removed or simplified.
Move long or repeated script sections into standalone scripts in the project.
Use parent and child pipelines to move some
work to jobs in an independent child pipeline.

On a self-managed instance, you can increase the size limits.

Error 500 when editing the `.gitlab-ci.yml` file

A loop of included configuration files
can cause a 500 error when editing the .gitlab-ci.yml file with the web editor.

Pipeline warnings

Pipeline configuration warnings are shown when you:

Validate configuration with the CI Lint tool.
Manually run a pipeline.

«Job may allow multiple pipelines to run for a single action» warning

When you use rules with a when clause without an if
clause, multiple pipelines may run. Usually this occurs when you push a commit to
a branch that has an open merge request associated with it.

To prevent duplicate pipelines, use
workflow: rules or rewrite your rules to control
which pipelines can run.

Console workaround if job using `resource_group` gets stuck (FREE SELF)

# find resource group by name
resource_group = Project.find_by_full_path('...').resource_groups.find_by(key: 'the-group-name')
busy_resources = resource_group.resources.where('build_id IS NOT NULL')

# identify which builds are occupying the resource
# (I think it should be 1 as of today)
busy_resources.pluck(:build_id)

# it's good to check why this build is holding the resource.
# Is it stuck? Has it been forcefully dropped by the system?
# free up busy resources
busy_resources.update_all(build_id: nil)

Job log slow to update

When you visit the job log page for a running job, there could be a delay of up to
60 seconds before the log updates. The default refresh time is 60 seconds, but after
the log is viewed in the UI, the following log updates should occur every 3 seconds.

Disaster recovery

You can disable some important but computationally expensive parts of the application
to relieve stress on the database during ongoing downtime.

Disable fair scheduling on shared runners

When clearing a large backlog of jobs, you can temporarily enable the ci_queueing_disaster_recovery_disable_fair_scheduling
feature flag. This flag disables fair scheduling
on shared runners, which reduces system resource usage on the jobs/request endpoint.

When enabled, jobs are processed in the order they were put in the system, instead of
balanced across many projects.

Disable CI/CD minutes quota enforcement

To disable the enforcement of CI/CD minutes quotas on shared runners, you can temporarily
enable the ci_queueing_disaster_recovery_disable_quota feature flag.
This flag reduces system resource usage on the jobs/request endpoint.

When enabled, jobs created in the last hour can run in projects which are out of quota.
Earlier jobs are already canceled by a periodic background worker (StuckCiJobsWorker).

CI/CD troubleshooting rails console commands

The following commands are run in the rails console.

WARNING:
Any command that changes data directly could be damaging if not run correctly, or under the right conditions.
We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case.

Cancel stuck pending pipelines

project = Project.find_by_full_path('<project_path>')
Ci::Pipeline.where(project_id: project.id).where(status: 'pending').count
Ci::Pipeline.where(project_id: project.id).where(status: 'pending').each {|p| p.cancel if p.stuck?}
Ci::Pipeline.where(project_id: project.id).where(status: 'pending').count

Try merge request integration

project = Project.find_by_full_path('<project_path>')
mr = project.merge_requests.find_by(iid: <merge_request_iid>)
mr.project.try(:ci_integration)

Validate the `.gitlab-ci.yml` file

project = Project.find_by_full_path('<project_path>')
content = p.repository.gitlab_ci_yml_for(project.repository.root_ref_sha)
Gitlab::Ci::Lint.new(project: project,  current_user: User.first).validate(content)

Disable AutoDevOps on Existing Projects

Project.all.each do |p|
  p.auto_devops_attributes={"enabled"=>"0"}
  p.save
end

Obtain runners registration token

Gitlab::CurrentSettings.current_application_settings.runners_registration_token

Seed runners registration token

appSetting = Gitlab::CurrentSettings.current_application_settings
appSetting.set_runners_registration_token('<new-runners-registration-token>')
appSetting.save!

Run pipeline schedules manually

You can run pipeline schedules manually through the Rails console to reveal any errors that are usually not visible.

# schedule_id can be obtained from Edit Pipeline Schedule page
schedule = Ci::PipelineSchedule.find_by(id: <schedule_id>)

# Select the user that you want to run the schedule for
user = User.find_by_username('<username>')

# Run the schedule
ps = Ci::CreatePipelineService.new(schedule.project, user, ref: schedule.ref).execute!(:schedule, ignore_skip_ci: true, save_on_errors: false, schedule: schedule)

How to get help

If you are unable to resolve pipeline issues, you can get help from:

The GitLab community forum
GitLab Support

Источник

I have been stuck for quite some time.
I wanted to use Gitlab for some personal projects and I am running it on my own server.

I just recently took interest in the CI/CD part of Gitlab and wanted to try it out.
However the runner keeps looping with a 500 internal server error.

WARNING: Checking for jobs... failed                runner=YmfuCQdb 
status=500 Internal Server Error
Feeding runners to channel                          builds=0
Dialing: tcp git.server.be:443 ...

I have googled this issue and cannot for the life of me solve it.

Some background information:
I migrated this server a while back, everything functioned just fine.
Any information you need I will try to provide.

Edit: I found the following in my logs [sanitized]

Started POST "/api/v4/jobs/request" for 0.0.0.0 at 2019-10-03 19:32:06 +0000

Processing by Gitlab::RequestForgeryProtection::Controller#index as JSON

Parameters: {"info"=>{"name"=>"gitlab-runner", "version"=>"12.3.0", "revision"=>"a8a019e0", "platform"=>"windows", "architecture"=>"amd64", "executor"=>"shell", "shell"=>"powershell", "features"=>{"variables"=>"[FILTERED]", "image"=>false, "services"=>false, "artifacts"=>true, "cache"=>true, "shared"=>true, "upload_multiple_artifacts"=>true, "upload_raw_artifacts"=>true, "session"=>false, "terminal"=>false, "refspecs"=>true, "masking"=>true, "proxy"=>false}}, "token"=>"[FILTERED]", "request_forgery_protection"=>{"info"=>{"name"=>"gitlab-runner", "version"=>"12.3.0", "revision"=>"a8a019e0", "platform"=>"windows", "architecture"=>"amd64", "executor"=>"shell", "shell"=>"powershell", "features"=>{"variables"=>"[FILTERED]", "image"=>false, "services"=>false, "artifacts"=>true, "cache"=>true, "shared"=>true, "upload_multiple_artifacts"=>true, "upload_raw_artifacts"=>true, "session"=>false, "terminal"=>false, "refspecs"=>true, "masking"=>true, "proxy"=>false}}, "token"=>"[FILTERED]"}}

Can't verify CSRF token authenticity.

This CSRF token verification failure is handled internally by `GitLab::RequestForgeryProtection`

Unlike the logs may suggest, this does not result in an actual 422 response to the user

For API requests, the only effect is that `current_user` will be `nil` for the duration of the request

Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms)

OpenSSL::Cipher::CipherError ():

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'

/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'

/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/base.rb:27:in `ensure_token'

/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:32:in `ensure_token'

/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable.rb:40:in `block in add_authentication_token_field'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:426:in `block in make_lambda'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:606:in `block (2 levels) in default_terminator'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:605:in `catch'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:605:in `block in default_terminator'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:199:in `block in halting'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:513:in `block in invoke_before'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:513:in `each'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:513:in `invoke_before'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:131:in `run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:816:in `_run_save_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/callbacks.rb:342:in `create_or_update'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/persistence.rb:275:in `save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/validations.rb:46:in `save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:310:in `block (2 levels) in save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:212:in `transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:310:in `block in save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:325:in `rollback_active_record_state!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:309:in `save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/suppressor.rb:44:in `save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:509:in `block in save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:528:in `block in around_save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:150:in `block in run_actions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in `catch_exceptions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in `run_actions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:133:in `run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:224:in `run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:128:in `block in run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:371:in `before'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:190:in `block in run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:292:in `block in pausable'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:292:in `catch'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:292:in `pausable'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:190:in `run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:127:in `run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:212:in `run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block (2 levels) in perform'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `catch'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block in perform'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:182:in `block in within_transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:171:in `block in within_transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:1868:in `block in within_transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:555:in `block in transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in `block in transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'

/opt/gitlab/embedded/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in `transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:212:in `transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:554:in `transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:1868:in `within_transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:170:in `within_transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:181:in `within_transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:62:in `perform'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:528:in `around_save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:509:in `save'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:154:in `block (2 levels) in run_actions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:154:in `each'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:154:in `block in run_actions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in `catch_exceptions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in `run_actions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:60:in `perform'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:163:in `perform'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:155:in `fire'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:219:in `block in add_actions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:729:in `block (2 levels) in define_helper'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:224:in `block in add_actions'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:729:in `block (2 levels) in define_helper'

/opt/gitlab/embedded/service/gitlab-rails/app/services/ci/register_job_service.rb:95:in `assign_runner!'

/opt/gitlab/embedded/service/gitlab-rails/app/services/ci/register_job_service.rb:50:in `block in execute'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/delegation.rb:71:in `each'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/delegation.rb:71:in `each'

/opt/gitlab/embedded/service/gitlab-rails/app/services/ci/register_job_service.rb:44:in `execute'

/opt/gitlab/embedded/service/gitlab-rails/lib/api/runner.rb:120:in `block (2 levels) in <class:Runner>'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `block (2 levels) in generate_api_method'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:170:in `instrument'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:56:in `block in generate_api_method'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:262:in `block in run'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:170:in `instrument'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:243:in `run'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:313:in `block in build_stack'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/resource.rb:20:in `_call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/resource/bearer.rb:8:in `_call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/abstract/handler.rb:17:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:38:in `block in call!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `catch'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `call!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `catch'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `call!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:227:in `call!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:221:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router/route.rb:72:in `exec'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:121:in `process_route'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:74:in `block in identity'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:93:in `transaction'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:72:in `identity'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:57:in `block in call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:137:in `with_optimization'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:56:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:119:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:45:in `call!'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:40:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/mapper.rb:48:in `serve'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:52:in `block in serve'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `each'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `serve'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:840:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/transaction.rb:57:in `run'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:17:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:103:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:16:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:20:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:13:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/correlation_id.rb:16:in `block in call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.3.0/lib/labkit/correlation/correlation_id.rb:18:in `use_id'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/correlation_id.rb:15:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/batch-loader-1.4.0/lib/batch_loader/middleware.rb:11:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/apollo_upload_server-2.0.0.beta.3/lib/apollo_upload_server/middleware.rb:20:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-attack-4.4.1/lib/rack/attack.rb:107:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `catch'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-cors-1.0.2/lib/rack/cors.rb:97:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/tempfile_reaper.rb:15:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/etag.rb:25:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/conditional_get.rb:38:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/http/content_security_policy.rb:18:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:40:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:18:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:232:in `context'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:226:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/cookies.rb:670:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:98:in `run_callbacks'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/basic_health_check.rb:25:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:38:in `call_app'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:26:in `block in call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `block in tagged'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:28:in `tagged'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `tagged'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:26:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_context.rb:26:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/request_id.rb:27:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/runtime.rb:22:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/executor.rb:14:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sentry-raven-2.9.0/lib/raven/integrations/rack.rb:51:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:524:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `public_send'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `method_missing'

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:12:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:606:in `process_client'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:701:in `worker_loop'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:142:in `start'

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/bin/unicorn:126:in `<top (required)>'

/opt/gitlab/embedded/bin/unicorn:23:in `load'

/opt/gitlab/embedded/bin/unicorn:23:in `<main>'

Источник

Here is what happen, i got either a 500 or 502 error after an upgrade using Omnibus method. And got the error message «Gitlab is not responding»

40TPT

The first thing i did was to look for what could have happen. And the first place to see that is on the log file located at

/var/log/gitlab/gitlab-rails/production.log

And the log basically just give me an error as shown below,

Started GET "/users/sign_in" for 175.144.6.68 at 2015-12-12 23:48:54 +0800
Processing by SessionsController#new as HTML
Completed 500 Internal Server Error in 98ms (ActiveRecord: 10.8ms)

NoMethodError (undefined method `push_events=' for #<GitlabCiService:0x0000000463dba8>):
  app/models/project.rb:809:in `builds_enabled='
  app/controllers/application_controller.rb:194:in `add_gon_variables'

But when i do a status check, it gives me this

[[email protected] gitlab-rails]# gitlab-ctl status
run: gitlab-workhorse: (pid 4934) 1009s; run: log: (pid 4147) 1227s
run: logrotate: (pid 4942) 1008s; run: log: (pid 296) 3434s
run: nginx: (pid 4948) 1008s; run: log: (pid 299) 3434s
run: postgresql: (pid 4957) 1007s; run: log: (pid 301) 3434s
run: redis: (pid 4965) 1007s; run: log: (pid 294) 3434s
run: sidekiq: (pid 4972) 1005s; run: log: (pid 302) 3434s
run: unicorn: (pid 4990) 1004s; run: log: (pid 305) 3434s

I have pretty much no idea what is going on. But after trying out different ways, it seems to boil down to the following,

1. Check what is going on

Firing the following command should give you an idea what is going on with your configure.

sudo gitlab-rake gitlab:check

After that you could try see what is causing it.

2. Forget to turn on postgres before upgrade

Well, because gitlab said to shutdown gitlab before upgrading, hence i did this,

gitlab-ctl stop

which stops everything including postgres. Hence, database migration wasn’t possible. Therefore, i fire the following command and see whether that helps

gitlab-rake db:migrate

Now, after this i still got a 502 error but at least i’m not stuck with 500 error!

3. Forget to reconfigure after an upgrade

Well, if its not database migration, then every time you did a migration, remember to do a reconfigure!

gitlab-ctl reconfigure

Once i did this. Wait a while, and puff! The screen is back up!

I’m just grateful everything is ok! Just remember to back up your VM image before doing all these upgrades!

Источник

Troubleshooting CI/CD (FREE)

Verify syntax

Edit .gitlab-ci.yml with the pipeline editor

Edit .gitlab-ci.yml locally

Verify syntax with CI Lint tool

Verify variables

GitLab CI/CD documentation

Documentation for pipeline types

Troubleshooting Guides for CI/CD features

Common CI/CD issues

Jobs or pipelines don’t run when expected

Two pipelines run at the same time

A job is not in the pipeline

No pipeline or the wrong type of pipeline runs

Pipeline with many jobs fails to start

A job runs unexpectedly

«fatal: reference is not a tree» error

Merge request pipeline messages

«Checking ability to merge automatically» message

«Checking pipeline status» message

Merge request status messages

«A CI/CD pipeline must run and be successful before merge» message

«Merge blocked: pipeline must succeed. Push a new commit that fixes the failure» message

Project group/project not found or access denied

«The parsed YAML is too big» message

Error 500 when editing the .gitlab-ci.yml file

Pipeline warnings

«Job may allow multiple pipelines to run for a single action» warning

Console workaround if job using resource_group gets stuck (FREE SELF)

Job log slow to update

Disaster recovery

Disable fair scheduling on shared runners

Disable CI/CD minutes quota enforcement

CI/CD troubleshooting rails console commands

Cancel stuck pending pipelines

Try merge request integration

Validate the .gitlab-ci.yml file

Disable AutoDevOps on Existing Projects

Obtain runners registration token

Seed runners registration token

Run pipeline schedules manually

How to get help

1. Check what is going on

2. Forget to turn on postgres before upgrade

3. Forget to reconfigure after an upgrade

Читайте также:

Edit `.gitlab-ci.yml` with the pipeline editor

Edit `.gitlab-ci.yml` locally

Project `group/project` not found or access denied

Error 500 when editing the `.gitlab-ci.yml` file

Console workaround if job using `resource_group` gets stuck (FREE SELF)

Validate the `.gitlab-ci.yml` file