gitlab runner在执行流水线作业时总是莫名奇妙的卡住不动。

本帖最后由 emailfromgt 于 2024-6-5 17:53 编辑

本帖最后由 emailfromgt 于 2024-6-5 17:49 编辑

本帖最后由 emailfromgt 于 2024-6-5 15:34 编辑

本帖最后由 emailfromgt 于 2024-6-5 15:25 编辑

GitLab 版本17.0.1

GitLab Runner 试过16.0.1 17.0.0版本，都会有同样的问题，

Runner 使用ssh，远程到AlmaLinux 9.0 (Emerald Puma)

我在另外的环境有一个同样部署的版本（GitLab 和GitLab-runner），只是服务器版本是CentOS ,用了5年都没出现过一次这样的问题。

有时候同一个.gitlab-ci.yml文件内容，前面还能执行，后面就不能执行

log界面显示如下内容。

Getting source from Git repository
#!/usr/bin/env bash
if set -o | grep pipefail > /dev/null; then set -o pipefail; fi; set -o errexit
set +o noclobber
: | eval $'export FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION=$\'false\'\nexport FF_NETWORK_PER_BUILD=$\'false\'\nexport FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=$\'false\'\nexport FF_USE_DIRECT_DOWNLOAD=$\'true\'\nexport FF_SKIP_NOOP_BUILD_STAGES=$\'true\'\nexport FF_USE_FASTZIP=$\'false\'\nexport FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR=$\'false\'\nexport FF_ENABLE_BASH_EXIT_CODE_CHECK=$\'false\'\nexport FF_USE_WINDOWS_LEGACY_PROCESS_STRATEGY=$\'true\'\nexport FF_USE_NEW_BASH_EVAL_STRATEGY=$\'false\'\nexport FF_USE_POWERSHELL_PATH_RESOLVER=$\'false\'\nexport FF_USE_DYNAMIC_TRACE_FORCE_SEND_INTERVAL=$\'false\'\nexport FF_SCRIPT_SECTIONS=$\'false\'\nexport FF_USE_NEW_SHELL_ESCAPE=$\'false\'\nexport FF_ENABLE_JOB_CLEANUP=$\'false\'\nexport FF_KUBERNETES_HONOR_ENTRYPOINT=$\'false\'\nexport FF_POSIXLY_CORRECT_ESCAPES=$\'false\'\nexport FF_USE_IMPROVED_URL_MASKING=$\'false\'\nexport FF_RESOLVE_FULL_TLS_CHAIN=$\'true\'\nexport FF_DISABLE_POWERSHELL_STDIN=$\'false\'\nexport FF_USE_POD_ACTIVE_DEADLINE_SECONDS=$\'false\'\nexport FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=$\'false\'\nexport FF_SET_PERMISSIONS_BEFORE_CLEANUP=$\'true\'\nexport FF_SECRET_RESOLVING_FAILS_IF_MISSING=$\'true\'\nexport CI_RUNNER_SHORT_TOKEN=$\'yg1ktX5PK\'\nexport CI_BUILDS_DIR=$\'builds\'\nexport CI_PROJECT_DIR=$\'builds/yg1ktX5PK/0/php/project_name\'\nexport CI_CONCURRENT_ID=0\nexport CI_CONCURRENT_PROJECT_ID=0\nexport CI_SERVER=$\'yes\'\nexport CI_JOB_STATUS=$\'running\'\nexport CI_JOB_TIMEOUT=3600\nexport CI_PIPELINE_ID=987\nexport CI_PIPELINE_URL=$\'http://git.domain.com/php/project_name/-/pipelines/987\'\nexport CI_JOB_ID=2927\nexport CI_JOB_URL=$\'http://git.domain.com/php/project_name/-/jobs/2927\'\nexport CI_JOB_TOKEN=$\'[MASKED]\'\nexport CI_JOB_STARTED_AT=$\'2024-06-05T15:15:47+08:00\'\nexport CI_REGISTRY_USER=$\'gitlab-ci-token\'\nexport CI_REGISTRY_PASSWORD=$\'[MASKED]\'\nexport CI_REPOSITORY_URL=$\'http://gitlab-ci-token:[MASKED]@git.domain.com/php/project_name.git\'\nexport CI_DEPENDENCY_PROXY_USER=$\'gitlab-ci-token\'\nexport CI_DEPENDENCY_PROXY_PASSWORD=$\'[MASKED]\'\nexport CI_JOB_NAME=$\'package fht\'\nexport CI_JOB_NAME_SLUG=$\'package-fht\'\nexport CI_JOB_STAGE=$\'package\'\nexport CI_JOB_MANUAL=$\'true\'\nexport CI_NODE_TOTAL=1\nexport CI=$\'true\'\nexport GITLAB_CI=$\'true\'\nexport CI_SERVER_FQDN=$\'git.domain.com\'\nexport CI_SERVER_URL=$\'http://git.domain.com\'\nexport CI_SERVER_HOST=$\'git.domain.com\'\nexport CI_SERVER_PORT=80\nexport CI_SERVER_PROTOCOL=$\'http\'\nexport CI_SERVER_SHELL_SSH_HOST=$\'git.domain.com\'\nexport CI_SERVER_SHELL_SSH_PORT=2222\nexport CI_SERVER_NAME=$\'GitLab\'\nexport CI_SERVER_VERSION=17.0.1\nexport CI_SERVER_VERSION_MAJOR=17\nexport CI_SERVER_VERSION_MINOR=0\nexport CI_SERVER_VERSION_PATCH=1\nexport CI_SERVER_REVISION=$\'bd824d1abb2\'\nexport GITLAB_FEATURES=\'\'\nexport CI_PROJECT_ID=121\nexport CI_PROJECT_NAME=$\'project_name\'\nexport CI_PROJECT_TITLE=$\'project_name\'\nexport CI_PROJECT_DESCRIPTION=\'\'\nexport CI_PROJECT_PATH=$\'php/project_name\'\nexport CI_PROJECT_PATH_SLUG=$\'php-project_name\'\nexport CI_PROJECT_NAMESPACE=$\'php\'\nexport CI_PROJECT_NAMESPACE_ID=2\nexport CI_PROJECT_ROOT_NAMESPACE=$\'php\'\nexport CI_PROJECT_URL=$\'http://git.domain.com/php/project_name\'\nexport CI_PROJECT_VISIBILITY=$\'internal\'\nexport CI_PROJECT_REPOSITORY_LANGUAGES=$\'php,javascript,smarty,css,html\'\nexport CI_PROJECT_CLASSIFICATION_LABEL=\'\'\nexport CI_DEFAULT_BRANCH=$\'master\'\nexport CI_CONFIG_PATH=$\'.gitlab-ci.yml\'\nexport CI_PAGES_DOMAIN=$\'example.com\'\nexport CI_PAGES_URL=$\'http://php.example.com/project_name\'\nexport CI_REGISTRY=$\'git.domain.com\'\nexport CI_REGISTRY_IMAGE=$\'git.domain.com/php/project_name\'\nexport CI_DEPENDENCY_PROXY_SERVER=$\'git.domain.com:80\'\nexport CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX=$\'git.domain.com:80/php/dependency_proxy/containers\'\nexport CI_[root@hostname ~]# #!/usr/bin/env bash
[root@hostname ~]# 
</dev/null; then set -o pipefail; fi; set -o errexit
[root@hostname ~]# set +o noclobber
<on >/dev/null 2>&1; then\n  $\'git\' lfs pull\n  ec

---

补充gitlab runner日志。其中14:24左右有一个执行卡住的作业。这是通过journalctl查看到的对应日志

Jun 05 14:22:09 hostname gitlab-runner[2207020]: Submitting job to coordinator...ok                  bytesize=5384 checksum=crc32:9c14d1f1 code=200 job=2887 job-status=running runner=gFmFUeGWG update-interval=0s
Jun 05 14:23:09 hostname gitlab-runner[2207020]: Updating job...                                     bytesize=5384 checksum=crc32:9c14d1f1 job=2887 runner=gFmFUeGWG
Jun 05 14:23:09 hostname gitlab-runner[2207020]: Submitting job to coordinator...ok                  bytesize=5384 checksum=crc32:9c14d1f1 code=200 job=2887 job-status=running runner=gFmFUeGWG update-interval=0s
Jun 05 14:23:39 hostname gitlab-runner[2207020]: Checking for jobs... received                       job=2888 repo_url=http://xxxxxxx.git runner=gFmFUeGWG
Jun 05 14:23:42 hostname gitlab-runner[2207020]: Appending trace to coordinator...ok                 code=202 job=2888 job-log=0-5384 job-status=running runner=gFmFUeGWG sent-log=0-5383 status=202 Accepted update-interval=1m0s
Jun 05 14:24:09 hostname gitlab-runner[2207020]: Updating job...                                     bytesize=5384 checksum=crc32:9c14d1f1 job=2887 runner=gFmFUeGWG
Jun 05 14:24:10 hostname gitlab-runner[2207020]: WARNING: Submitting job to coordinator... job failed  bytesize=5384 checksum=crc32:9c14d1f1 code=403 job=2887 job-status= runner=gFmFUeGWG update-interval=0s
Jun 05 14:24:10 hostname gitlab-runner[2207020]: WARNING: Error while executing file based variables removal script  error=context canceled job=2887 project=66 runner=gFmFUeGWG
Jun 05 14:24:10 hostname gitlab-runner[2207020]: WARNING: Job failed: canceled
Jun 05 14:24:10 hostname gitlab-runner[2207020]:                       duration_s=303.348894709 job=2887 project=66 runner=gFmFUeGWG
Jun 05 14:24:10 hostname gitlab-runner[2207020]: WARNING: Appending trace to coordinator... job failed  code=403 job=2887 job-log= job-status= runner=gFmFUeGWG sent-log=5384-5463 status=403 Forbidden update-interval=0s
Jun 05 14:24:10 hostname gitlab-runner[2207020]: Updating job...                                     bytesize=5464 checksum=crc32:b382a911 job=2887 runner=gFmFUeGWG
Jun 05 14:24:10 hostname gitlab-runner[2207020]: WARNING: Submitting job to coordinator... job failed  bytesize=5464 checksum=crc32:b382a911 code=403 job=2887 job-status= runner=gFmFUeGWG update-interval=0s
Jun 05 14:24:10 hostname gitlab-runner[2207020]: WARNING: Failed to process runner                   builds=1 error=canceled executor=ssh runner=gFmFUeGWG
Jun 05 14:24:42 hostname gitlab-runner[2207020]: Updating job...                                     bytesize=5384 checksum=crc32:58bd86ed job=2888 runner=gFmFUeGWG
Jun 05 14:24:42 hostname gitlab-runner[2207020]: Submitting job to coordinator...ok                  bytesize=5384 checksum=crc32:58bd86ed code=200 job=2888 job-status=running runner=gFmFUeGWG update-interval=0s
Jun 05 14:25:42 hostname gitlab-runner[2207020]: Updating job...                                     bytesize=5384 checksum=crc32:58bd86ed job=2888 runner=gFmFUeGWG
Jun 05 14:25:42 hostname gitlab-runner[2207020]: Submitting job to coordinator...ok                  bytesize=5384 checksum=crc32:58bd86ed code=200 job=2888 job-status=running runner=gFmFUeGWG update-interval=0s
Jun 05 14:26:42 hostname gitlab-runner[2207020]: Updating job...                                     bytesize=5384 checksum=crc32:58bd86ed job=2888 runner=gFmFUeGWG
Jun 05 14:26:42 hostname gitlab-runner[2207020]: Submitting job to coordinator...ok                  bytesize=5384 checksum=crc32:58bd86ed code=200 job=2888 job-status=running runner=gFmFUeGWG update-interval=0s
Jun 05 14:27:42 hostname gitlab-runner[2207020]: Updating job...                                     bytesize=5384 checksum=crc32:58bd86ed job=2888 runner=gFmFUeGWG
Jun 05 14:27:43 hostname gitlab-runner[2207020]: Submitting job to coordinator...ok                  bytesize=5384 checksum=crc32:58bd86ed code=200 job=2888 job-status=running runner=gFmFUeGWG update-interval=0s
Jun 05 14:27:54 hostname gitlab-runner[2207020]: Checking for jobs... received                       job=2889 repo_url=http://xxxxxxx.git runner=gFmFUeGWG
Jun 05 14:27:58 hostname gitlab-runner[2207020]: Appending trace to coordinator...ok                 code=202 job=2889 job-log=0-5384 job-status=running runner=gFmFUeGWG sent-log=0-5383 status=202 Accepted update-interval=3s
Jun 05 14:28:28 hostname gitlab-runner[2207020]: Updating job...                                     bytesize=5384 checksum=crc32:e26c3f45 job=2889 runner=gFmFUeGWG
Jun 05 14:28:28 hostname gitlab-runner[2207020]: Submitting job to coordinator...ok                  bytesize=5384 checksum=crc32:e26c3f45 code=200 job=2889 job-status=running runner=gFmFUeGWG update-interval=0s
Jun 05 14:28:43 hostname gitlab-runner[2207020]: Updating job...                                     bytesize=5384 checksum=crc32:58bd86ed job=2888 runner=gFmFUeGWG
Jun 05 14:28:43 hostname gitlab-runner[2207020]: WARNING: Submitting job to coordinator... job failed  bytesize=5384 checksum=crc32:58bd86ed code=403 job=2888 job-status= runner=gFmFUeGWG update-interval=0s
Jun 05 14:28:43 hostname gitlab-runner[2207020]: WARNING: Error while executing file based variables removal script  error=context canceled job=2888 project=66 runner=gFmFUeGWG
Jun 05 14:28:43 hostname gitlab-runner[2207020]: WARNING: Job failed: canceled
Jun 05 14:28:43 hostname gitlab-runner[2207020]:                       duration_s=303.302740002 job=2888 project=66 runner=gFmFUeGWG
Jun 05 14:28:43 hostname gitlab-runner[2207020]: WARNING: Appending trace to coordinator... job failed  code=403 job=2888 job-log= job-status= runner=gFmFUeGWG sent-log=5384-5463 status=403 Forbidden update-interval=0s
Jun 05 14:28:43 hostname gitlab-runner[2207020]: Updating job...                                     bytesize=5464 checksum=crc32:256bb993 job=2888 runner=gFmFUeGWG
Jun 05 14:28:43 hostname gitlab-runner[2207020]: WARNING: Submitting job to coordinator... job failed  bytesize=5464 checksum=crc32:256bb993 code=403 job=2888 job-status= runner=gFmFUeGWG update-interval=0s
Jun 05 14:28:43 hostname gitlab-runner[2207020]: WARNING: Failed to process runner                   builds=1 error=canceled executor=ssh runner=gFmFUeGWG
Jun 05 14:28:58 hostname gitlab-runner[2207020]: Updating job...                                     bytesize=5384 checksum=crc32:e26c3f45 job=2889 runner=gFmFUeGWG

从报错来看，像是你的命令被截断了，命令运行失败了。检查一下 ci.yaml文件里面的命令直接到对应的机器能正常执行吗

转身 发表于 2024-6-5 17:26
从报错来看，像是你的命令被截断了，命令运行失败了。检查一下 ci.yaml文件里面的命令直接到对应的机器能正 ...

所有的命令都直接在机器上验证过，而且也没有复杂逻辑的命令。有时候修改完再执行就卡住，不取消就直到超时。有时候是没有修改ci.yml提交一两次代码再执行也会卡住。日志界面也没有可以分析的错误信息

有没有job的执行日志， 卡主时候的状态可能更好判断一点

转身 发表于 2024-6-6 11:11
有没有job的执行日志， 卡主时候的状态可能更好判断一点

第一次附的就是job的执行过程窗口显示的日志，最新行的内容就显示出类似“<on >/dev/null 2>&1; then\n  $\'git\' lfs pull\n  ec”     “<mFUeGWG/0/java/v3/project_name.tmp/git-templa”这样的就不动了，没有任何新的输出，直到超时

[quote][size=2][color=#999999]转身 发表于 2024-6-6 11:11[/color][/size] 有没有job的执行日志， 卡主时候的状态可能更好判断一点[/quote]