Open
Issue created Aug 23, 2019 by
Random error 500 when pulling/pushing to repository using HTTPS
EDIT: It also happens with pulls.
Summary
Just upgraded to 12.2 and getting random 500 errors when using git and pulling/pushing to a project using HTTPS.
Steps to reproduce
Try to pull any repository using HTTPS on http://lab.shelter.moe (self-hosted gitlab) or push to one you created there.
What is the current bug behavior?
Git commands fail with a 500 error. These errors are random as, when you try pushing again, and again, it sometimes fails during the push, and sometimes the push succeeds.
See this terminal log for an example :
$ git push -u origin master
fatal: unable to access 'https://lab.shelter.moe/axelterizaki/ultrastar2ass.git/': The requested URL returned error: 500
$ git push -u origin master
fatal: unable to access 'https://lab.shelter.moe/axelterizaki/ultrastar2ass.git/': The requested URL returned error: 500
$ git push -u origin master
Counting objects: 24, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (19/19), done.
Writing objects: 100% (24/24), 25.73 KiB | 0 bytes/s, done.
Total 24 (delta 1), reused 0 (delta 0)
error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500 Internal Server Error
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date
$ git push -u origin master
fatal: unable to access 'https://lab.shelter.moe/axelterizaki/ultrastar2ass.git/': The requested URL returned error: 500
$ git push -u origin master
fatal: unable to access 'https://lab.shelter.moe/axelterizaki/ultrastar2ass.git/': The requested URL returned error: 500
$ git push -u origin master
Counting objects: 24, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (19/19), done.
Writing objects: 100% (24/24), 25.73 KiB | 0 bytes/s, done.
Total 24 (delta 1), reused 0 (delta 0)
To https://lab.shelter.moe/axelterizaki/ultrastar2ass.git
* [new branch] master -> master
Branch master set up to track remote branch master from origin.What is the expected correct behavior?
It should be working all right as in 12.1.x
This happened ever since I upgraded Omnibus gitlab to 12.2 this morning.
Relevant logs and/or screenshots
I examined the production.log and here’s what I found when trying a git push :
Started GET "/axelterizaki/ultrastar2ass.git/info/refs?service=git-receive-pack" for 192.168.122.1 at 2019-08-23 12:55:18 +0200
Processing by Gitlab::RequestForgeryProtection::Controller#index as */*
Parameters: {"service"=>"git-receive-pack"}
Completed 200 OK in 0ms (ActiveRecord: 0.0ms)
Redis::CommandError (ERR value is not an integer or out of range):
lib/peek/views/redis_detailed.rb:10:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'Here it says it returned a 200, but my Apache frontend server returns a 500 instead :
xx.xxx.xx.xxx - - [23/Aug/2019:13:03:05 +0200] "GET /axelterizaki/ultrastar2ass.git/info/refs?service=git-receive-pack HTTP/1.1" 500 3228 "-" "git/2.19.0.windows.1"I’m not sure what’s wrong but I guess the Redis error in production.log has something to do with it.
Output of checks
See below
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 16.04 Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.22.0 Sidekiq Version:5.2.7 Go Version: unknown
GitLab information Version: 12.2.0 Revision: 1c1d47c5974 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: https://lab.shelter.moe HTTP Clone URL: https://lab.shelter.moe/some-group/some-project.git SSH Clone URL: git@shelter.mahoro-net.org:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: gitlab, google_oauth2, twitter, github
GitLab Shell Version: 9.3.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... yes Init.d configured correctly? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 5/2 ... yes 2/4 ... yes 1/6 ... yes 1/7 ... yes 1/8 ... yes 7/9 ... yes 19/10 ... yes 4/11 ... yes 22/12 ... yes 156/13 ... yes 15/14 ... yes 8/15 ... yes 5/18 ... yes 45/22 ... yes 156/23 ... yes 52/24 ... yes 52/25 ... yes 59/26 ... yes 155/28 ... yes 62/32 ... yes 62/33 ... yes 52/34 ... yes 155/35 ... yes 5/36 ... yes 54/40 ... yes 67/41 ... yes 67/43 ... yes 67/44 ... yes 156/45 ... yes 65/46 ... yes 66/47 ... yes 82/52 ... yes 96/55 ... yes 82/59 ... yes 8/65 ... yes 7/66 ... yes 5/67 ... yes 5/68 ... yes 82/69 ... yes 20/70 ... yes 45/71 ... yes 110/72 ... yes 82/73 ... yes 8/74 ... yes 8/75 ... yes 8/76 ... yes 156/77 ... yes 9/80 ... yes 66/81 ... yes 78/83 ... yes 1/87 ... yes 135/89 ... yes 126/90 ... yes 135/91 ... yes 135/92 ... yes 135/93 ... yes 1/98 ... yes 153/99 ... yes 155/100 ... yes 155/101 ... yes 5/102 ... yes 20/104 ... yes 8/106 ... yes 45/107 ... yes 102/109 ... yes 129/110 ... yes 5/111 ... yes 9/112 ... yes 198/113 ... yes 102/114 ... yes 198/115 ... yes 193/116 ... yes 202/117 ... yes 45/118 ... yes 135/119 ... yes 107/120 ... yes 107/121 ... yes 20/123 ... yes 5/129 ... yes 135/130 ... yes 5/131 ... yes 52/132 ... yes 1/133 ... yes 1/134 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.22.0 ? ... yes (2.22.0) Git user has default SSH configuration? ... yes Active users: ... 196
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
For now my only possible workaround is to migrate back to an older version (12.1) where this didn’t happen.
Edited Aug 23, 2019 by Guillaume Lebigot
We currently use SubVersion but are looking to migrate to a Git-based solution in order to be able to carry out pre-commit code reviews.
The requirements are that the central Git repository is hosted on-premises, has a visual front-end to allow management of projects, and uses Active Directory authentication.
As a trial, I have installed GitLab EE on a virtual machine running Ubuntu server 18.04.
I have set up AD authentication using the following config:
gitlab_rails['ldap_enabled'] = true
gitlab_rails['ldap_servers'] = YAML.load <<-'EOS'
main:
label: 'MyOrganisation'
host: '172.16.0.6'
port: 389
uid: 'sAMAccountName'
bind_dn: 'CN=ldapbinduser,CN=Users,DC=myorganisation,DC=com'
password: 'password'
timeout: 30
active_directory: true
allow_username_or_mail_login: false
lowercase_usernames: true
block_auto_created_users: true
base: 'OU=Software,OU=Engineering,DC=myorganisation,DC=com'
group_base: 'OU=Software,OU=Engineering,DC=myorganisation,DC=com'
admin_group: 'internal software dept'
EOS
This has been partially successful. A MyOrganisation tab appears in the GitLab logon page and members of the Software group are able to log on using their AD logon and password while non-members are not. Members of the “Internal Software Dept” group are not automatically assigned admin rights, but that’s fine because we can add them manually.
The problem comes when trying to push the history of an SVN repo into GitLab, or clone a repository. I have created an Internal Software group in GitLab and added a TestProject project to it with a readme.txt file. On my Windows 10 PC I have installed the Git Credential Manager for Windows. When I attempt to clone this to my PC using:
git clone http://gitlab/internal-software/testproject.git
I receive a message of :
fatal: unable to access ‘http://gitlab/internal-software/testproject.git/’: The requested URL returned error: 500
WireShark shows two attempts to connect with a 401 – Unauthorized response to the first, followed by a 500 – Internal Server Error response to the second:
1 local IP gitlab IP TCP 66 59710 → 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1
2 gitlab IP local IP TCP 66 80 → 59710 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128
3 local IP gitlab IP TCP 54 59710 → 80 [ACK] Seq=1 Ack=1 Win=525568 Len=0
4 local IP gitlab IP HTTP 241 GET /internal-software/testproject.git/info/refs?service=git-upload-pack HTTP/1.1
5 gitlab IP local IP TCP 60 80 → 59710 [ACK] Seq=1 Ack=188 Win=30336 Len=0
6 gitlab IP local IP HTTP 618 HTTP/1.1 401 Unauthorized (text/plain)
7 local IP gitlab IP TCP 54 59710 → 80 [ACK] Seq=188 Ack=565 Win=524800 Len=0
8 local IP gitlab IP HTTP 292 GET /internal-software/testproject.git/info/refs?service=git-upload-pack HTTP/1.1
9 gitlab IP local IP TCP 60 80 → 59710 [ACK] Seq=565 Ack=426 Win=31360 Len=0
10 gitlab IP local IP TCP 1514 80 → 59710 [ACK] Seq=565 Ack=426 Win=31360 Len=1460 [TCP segment of a reassembled PDU]
11 gitlab IP local IP TCP 1514 80 → 59710 [ACK] Seq=2025 Ack=426 Win=31360 Len=1460 [TCP segment of a reassembled PDU]
12 gitlab IP local IP HTTP 309 HTTP/1.1 500 Internal Server Error (text/html)
13 local IP gitlab IP TCP 54 59710 → 80 [ACK] Seq=426 Ack=3740 Win=525568 Len=0
14 local IP gitlab IP TCP 54 59710 → 80 [RST, ACK] Seq=426 Ack=3740 Win=0 Len=0
The content of the 500 – Internal Server Error response is the standard GitLab 500 – “Whoops, something went wrong on our end” page.
On the server, I went through the .log files in var/logs/gitlab/gitlab-rails and found this in production_json.log:
{«method»:»GET»,»path»:»/internal-software/testproject/git/info/refs»,»format»:»/«,»controller»:»Projects::GitHttpController»,»action»:»info_refs»,»status»:401,»duration»:35.81,»view»:1.05,»db»:9.7,»time»:»2019-09-19T08:37:55.371Z»,»params»:[{«key»:»service»,»value»:git-upload-pack»},{«key»:»namespage_id»,»value»:»internal-software»},{«key»:»project_id»,»value»:»testproject.git»}],»remote_ip»:»172.16.1.46″,»user_id»:null,»username»:null,»ua»:git/2.17.0.windows.1″,»queue_duration»:null,»correlation_id»:»long_uid»}
{«method»:»GET»,»path»:»/internal-software/testproject/git/info/refs»,»format»:»/«,»controller»:»Projects::GitHttpController»,»action»:»info_refs»,»status»:500,»error»:»ArgumentError:
encryption or method MUST be
provided»,»duration»:215.3,»view»:0.0,»db»:14.11,»time»:»2019-09-19T08:37:55.803Z»,»params»:[{«key»:»service»,»value»:git-upload-pack»},{«key»:»namespage_id»,»value»:»internal-software»},{«key»:»project_id»,»value»:»testproject.git»}],»remote_ip»:»172.16.1.46″,»user_id»:null,»username»:null,»ua»:git/2.17.0.windows.1″,»queue_duration»:null,»correlation_id»:»long_uid»}
These would seem to match up to the HTTP requests and responses above. The first is the 401 and the second is the 500.
The error message is ArgumentError: encryption or method MUST be provided
I have tried searching for this on the GitLab site, Stack Overflow, Stack Exchange and some well-known search engines, but only get approximate results that don’t match my problem.
Вопрос:
Я не могу клонировать недавно созданный репозиторий. Я становлюсь ниже ошибки.
$ git clone https://github.xxxxx.com/zzzzzz.git
Cloning into 'zzzzzz'...
Username for 'https://github.xxxxxx.com': yyyyy
Password for 'https://yyyyy@github.xxxxxx.com':
remote: Internal Server Error.
remote:
fatal: unable to access 'https://github.xxxxxx.com/zzzzz.git/': The requested URL returned error: 500
Я успешно сгенерировал ключи ssh и обновил ключ в настройках github в соответствии с инструкциями в приведенном ниже URL
https://help.github.com/articles/generating-ssh-keys/
аутентификация была успешной, когда я выполнил команду:
git -T git @github.xxx.com
Также, как я понимаю, если мы установим ssh, команда git clone не должна запрашивать имя пользователя и пароль. Но он все еще просит их.
debug1: Authentication succeeded (publickey).
Authenticated to github.xxxxx.com ([10.28.22.44]:22).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
Hi xxxx! You've successfully authenticated, but GitHub does not provide shell access.
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 3880, received 1696 bytes, in 0.2 seconds
Bytes per second: sent 19132.2, received 8363.0
debug1: Exit status 1
Лучший ответ:
Это, скорее всего, проблема с сервисом GitHub. Я рекомендую связаться с ними и объяснить, в чем проблема.
Вот как Wikipedia объясняет код ответа 500:
Содержание
- 500 Внутренняя ошибка сервера
- У меня такая же проблема, но в gitlab, и ответ на проблему заключается в том, что gitlab делает развертывание
- Если эта проблема не устранена, обратитесь к администратору GitLab.
500 Внутренняя ошибка сервера
Общее сообщение об ошибке, данное при неожиданном условии встречается, и не более конкретное сообщение подходит.
В принципе, что-то пошло не так на конечной точке GitHub.
Но, поскольку вы уже установили свои SSH-ключи, вы можете использовать ssh url для клонирования вашего репозитория:
git clone git@github.com:owner/repo.git
Ответ №1
У меня такая же проблема, но в gitlab, и ответ на проблему заключается в том, что gitlab делает развертывание
Выполняется развертывание
Повторите попытку через несколько минут.
Если эта проблема не устранена, обратитесь к администратору GitLab.
поэтому я был связан через несколько часов, и он работает.
Иногда возникает проблема при выполнении merge requests в gitlab.
В моем случае, у меня был старый gitlab, который пришлось обновить с версии 6.3 до версии 10.х (а в конечном итоге и до самой актуальной).
Ошибка имеет следующий вид:
|
Processing by Projects::MergeRequests::CreationsController#create as HTML Parameters: {«utf8»=>«<E2><9C><93>», «authenticity_token»=>«[FILTERED]», «merge_request»=>{«title»=>«Sonarqube Helm», «description»=>«», «assignee_ids»=>[«0»], «label_ids»=>[«»], «force_remove_source_branch»=>«0», «squash»=>«0», «lock_version»=>«0», «source_project_id»=>«332», «source_branch»=>«sonarqube», «target_project_id»=>«332», «target_branch»=>«master»}, «namespace_id»=>«roman», «project_id»=>«infrastructure»} Completed 500 Internal Server Error in 2050ms (ActiveRecord: 132.4ms) Started POST «/api/v4/jobs/request» for 192.168.1.2 at 2019—09—19 09:37:00 +0000 ActiveRecord::StatementInvalid (Mysql2::Error: Incorrect string value: ‘xF0x9Fx94x90mT…’ for column ‘diff’ at row 197: INSERT INTO merge_request_diff_files (`diff`, `new_path`, `old_path`, `a_mode`, `b_mode`, `new_file`, `renamed_file`, `deleted_file`, `too_large`, `binary`, `merge_request_diff_id`, `relative_order`) VALUES (‘@@ -0,0 +1,23 @@n+# Patterns to ignore when building packages.n+# This supports shell glob matching, relative path matching, andn+# negation (prefixed with !). Only one pattern per line.n+.DS_Storen+# Common VCS dirsn+.git/n+.gitignoren+.bzr/n+.bzrignoren+.hg/n+.hgignoren+.svn/n+# Common backup filesn+*.swpn+*.bakn+*.tmpn+*~n+# Various IDEsn+.projectn+.idea/n+*.tmprojn+# OWNERS file for Kubernetesn+OWNERSn’, ‘helm/sonarqube/.helmignore’, ‘helm/sonarqube/.helmignore’, ‘0’, ‘100644’, 1, 0, 0, 0, 0, 5050, 0), (‘@@ —0,0 +1,19 @@n+apiVersion: v1n+name: sonarquben+description: Sonarqube is an open sourced code quality scanning tooln+version: 2.3.0n+appVersion: 7.9n+keywords:n+ — coveragen+ — securityn+ — coden+ — qualityn+home: https://www.sonarqube.org/n+icon: https://www.Started POST «/api/v4/jobs/request» for 192.168.1.2 at 2019—09—19 09:36:54 +0000 |
Причина кроется в размере выделяемой памяти для той или иной кодировки https://dev.mysql.com/doc/refman/5.5/en/charset-charsets.html
Проверять необходимо всю базу, но начать можно с табличек в базе gitlabhq_production с префиксами «merge_»:
|
merge_request_diff_commits merge_request_diff_files merge_request_diffs merge_request_metrics merge_requests merge_requests_closing_issues |
Кодировка данных таблиц была latin-swedish-ci, после конвертации в utf8mb4_general_ci проблемы с merge requests были исправлены.
Выведем список всех таблиц и сделаем выборку тех, которые не в кодировке utf8mb4_general_ci:
В качестве заметки привожу SQL запросы. Необходимо пройти по всем таблицам в базе и установить нужную кодировку.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
DESCRIBE merge_request_diff_commits; ALTER DATABASE `gitlabhq_production` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; USE gitlabhq_production; ALTER TABLE `gitlabhq_production`.`merge_request_diff_commits` CHANGE COLUMN `committed_date` `committed_date` TIMESTAMP NULL DEFAULT NULL ; ALTER TABLE `gitlabhq_production`.`merge_request_diff_commits` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; ALTER TABLE `gitlabhq_production`.`merge_request_diff_files` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; ALTER TABLE `gitlabhq_production`.`merge_request_diffs` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; ALTER TABLE `gitlabhq_production`.`merge_request_metrics` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; ALTER TABLE `gitlabhq_production`.`merge_requests` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; ALTER TABLE `gitlabhq_production`.`merge_requests_closing_issues` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; |
Иногда возникает ошибка вида:
|
Invalid default value for ‘updated_at’ |
или
|
Invalid default value for ‘created_at’ |
В данном случаи выполняем для проблемной таблицы запрос:
|
set sql_mode =»; ALTER TABLE approval_project_rules CHANGE COLUMN `created_at` `created_at` TIMESTAMP NULL DEFAULT NULL ; set sql_mode =»; ALTER TABLE approval_project_rules CHANGE COLUMN `updated_at` `updated_at` TIMESTAMP NULL DEFAULT NULL ; |
Также устанавливаем кодировку для всей базы gitlab:
|
ALTER DATABASE gitlabhq_production CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; |
Проверяем, что все ок
|
SELECT SCHEMA_NAME ‘gitlabhq_production’, default_character_set_name ‘charset’, DEFAULT_COLLATION_NAME ‘collation’ FROM information_schema.SCHEMATA; +———————————+—————+——————————+ | gitlabhq_production | charset | collation | +———————————+—————+——————————+ | ... | ... | ... | | gitlabhq_production | utf8mb4 | utf8mb4_general_ci | | ... | ... | ... | +———————————+—————+——————————+ |
Проверяем чтобы в настройках gitlab в конфиге также были указанные нужные нам кодировки:
|
cat config/database.yml # # PRODUCTION # production: adapter: mysql2 encoding: utf8mb4 collation: utf8mb4_general_ci reconnect: false database: gitlabhq_production host: mysql port: 3306 username: username password: «Password» pool: 10 |