your database configuration to ensure the username/password are valid.

1. 使用 k8s 1.26.2 的 Deployment 部署的 gitlab/gitlab-ce:16.2.4-ce.0，使用 local PV 持久化

2. gitlab 服务器今天由于资源不足，导致 gitlab 宕机

3. 机器资源充足时，启动 gitlab 时，出现异常日志，导致 gitlab 一直重启

   Running handlers:
    [2024-04-29T13:41:10+08:00] INFO: Running report handlers
    Running handlers complete
    [2024-04-29T13:41:10+08:00] INFO: Report handlers complete
    Infra Phase complete, 342/1547 resources updated in 02 minutes 08 seconds
    gitlab Reconfigured!
    Checking for unmigrated data on legacy storage
   
    Upgrade failed. Could not check for unmigrated data on legacy storage.
   
    Waiting until database is ready before continuing... Waiting until database is ready before continuing... Waiting until database is ready before continuing... Waiting until database is ready before continuing... Waiting until database is ready before continuing... Failed to connect to the database... Error: There is an issue connecting to your database with your username/password, username: gitlab. Please check your database configuration to ensure the username/password are valid.
   
    If you would like to restart the instance without performing this
    check, add the following to your docker command:
    -e GITLAB_SKIP_UNMIGRATED_DATA_CHECK=true

4. 给 Deployment 增加环境变量后，容器可以启动，但是 gitlab 的 postgresql 无法正常运行

5. 查看日志可知如下错入：

   ==> /var/log/gitlab/puma/current <==
    2024-04-29_07:04:45.06408 {"timestamp":"2024-04-29T07:04:45.063Z","pid":15594,"message":"! Unable to load application: ActiveRecord::DatabaseConnectionError: There is an issue connecting to your database with your username/password, username: gitlab.\n\nPlease check your database configuration to ensure the username/password are valid.\n"}
    2024-04-29_07:04:45.06426 bundler: failed to load command: puma (/opt/gitlab/embedded/bin/puma)
    2024-04-29_07:04:45.06448 /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:83:in `rescue in new_client': There is an issue connecting to your database with your username/password, username: gitlab. (ActiveRecord::DatabaseConnectionError)
    2024-04-29_07:04:45.06450 
    2024-04-29_07:04:45.06451 Please check your database configuration to ensure the username/password are valid.
    2024-04-29_07:04:45.06451   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:77:in `new_client'
    2024-04-29_07:04:45.06451   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:37:in `postgresql_connection'
    2024-04-29_07:04:45.06451   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:656:in `public_send'
    2024-04-29_07:04:45.06452   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:656:in `new_connection'
    2024-04-29_07:04:45.06452   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:700:in `checkout_new_connection'
    2024-04-29_07:04:45.06452   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:679:in `try_to_checkout_new_connection'
    2024-04-29_07:04:45.06452   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:640:in `acquire_connection'
    2024-04-29_07:04:45.06452   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:341:in `checkout'
    2024-04-29_07:04:45.06453   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `connection'
    2024-04-29_07:04:45.06453   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/connection_handler.rb:211:in `retrieve_connection'
    2024-04-29_07:04:45.06453   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_handling.rb:313:in `retrieve_connection'
    2024-04-29_07:04:45.06454   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_handling.rb:280:in `connection'
    2024-04-29_07:04:45.06454   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/model_schema.rb:407:in `table_exists?'
    2024-04-29_07:04:45.06454   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/attribute_methods/primary_key.rb:95:in `get_primary_key'
    2024-04-29_07:04:45.06454   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/attribute_methods/primary_key.rb:83:in `reset_primary_key'
    2024-04-29_07:04:45.06455   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/attribute_methods/primary_key.rb:71:in `primary_key'
    2024-04-29_07:04:45.06455   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/relation/delegation.rb:93:in `primary_key'
    2024-04-29_07:04:45.06455   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/relation/finder_methods.rb:575:in `ordered_relation'
    2024-04-29_07:04:45.06455   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/relation/finder_methods.rb:174:in `last'
    2024-04-29_07:04:45.06456   from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/querying.rb:22:in `last'

gitlab 在今天凌晨还可以备份数据

gitlab 的配置文件 gitlab-secrets.json、gitlab.rb 与历史配置文件已对比，无修改

已尝试在添加环境变量

1. GITLAB_SKIP_UNMIGRATED_DATA_CHECK=true

复制代码

后，在容器内执行

1. gitlab-ctl reconfigure
   
2. gitlab-ctl restart

复制代码

也是无法正常启动 postgresql

这个错误有可能是数据库连接数不足导致的，建议在 /etc/gitlab/gitlab.rb 中调整数据库连接数：

postgresql['max_connections'] = 500

# 具体多少根据环境自行调整

ffli 发表于 2024-4-29 15:35
这个错误有可能是数据库连接数不足导致的，建议在 /etc/gitlab/gitlab.rb 中调整数据库连接数：

postgresq ...

不行，

我将备份数据拷贝到其他机器上，在其他机器上安装 gitlab 后，可以正常恢复数据，没有任何问题。

在容器中看看 /var/log/gitlab/postgresql/current 日志报什么错误没有？

徐晓伟 发表于 2024-4-29 16:43
我将备份数据拷贝到其他机器上，在其他机器上安装 gitlab 后，可以正常恢复数据，没有任何问题。 ...

恢复数据时，使用到了 gitlab-secrets.json、gitlab.rb 文件，恢复过程完全正常，项目可以正常访问，流水线日志也可以正常访问。

如果 gitlab-secrets.json 配置文件不正确，将会导致 流水线日志打开报错。

ffli 发表于 2024-4-29 21:00
在容器中看看 /var/log/gitlab/postgresql/current 日志报什么错误没有？

没有日志