[分享]为Omnibus Gitlab配置独立的Gitaly集群

快速为Omnibus Gitlab配置独立的Gitaly集群

配置目标

给ominbus安装方式的Gitalb快速配置一个单praefect节点的Gitaly集群

准备阶段

你需要配置的清单

  • 1 个 腾讯云PostgreSQL 节点:10.0.0.40
  • 至少 1 个 Praefect 节点(需要较少的存储空间):10.0.10.35
  • 3 个 Gitaly 节点(高 CPU、高内存、快速存储)
    • gitaly-1:10.0.10.32
    • gitaly-2:10.0.10.45
    • gitaly-3:10.0.10.33
  • 1 个 GitLab 服务器:10.0.10.31

自定义时间服务器设置

您必须确保每个服务器节点的时间必须一致,否则会出现错误。

数据库

手动数据库设置

  1. 申请一台腾讯云PostgreSQL节点

  2. 创建一个新用户praefect供 Praefect 使用:

    CREATE ROLE praefect WITH LOGIN PASSWORD 'PRAEFECT_SQL_PASSWORD';
    

    PRAEFECT_SQL_PASSWORD:Praefect 使用此密码连接到 PostgreSQL。

  3. 创建一个praefect_production新数据库由praefect用户拥有。

    CREATE DATABASE praefect_production WITH OWNER praefect ENCODING UTF8;
    
  4. 配置praefect

    praefect['database_host'] = '10.0.0.40'
    praefect['database_port'] = 5432
    praefect['database_user'] = 'praefect'
    praefect['database_password'] = 'PRAEFECT_SQL_PASSWORD'
    praefect['database_dbname'] = 'praefect_production
    praefect['database_direct_host'] = '10.0.0.40'
    praefect['database_direct_port'] = 5432
    

配置praefect

在Praefect节点上:

  1. 通过编辑/etc/gitlab/gitlab.rb

    # external_url 'http://gitlab.example.com' # 此配置需要注释
    gitaly['enable'] = false
    postgresql['enable'] = false
    redis['enable'] = false
    nginx['enable'] = false
    puma['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
    prometheus['enable'] = false
    alertmanager['enable'] = false
    grafana['enable'] = false
    gitlab_exporter['enable'] = false
    gitlab_kas['enable'] = false
    
    # Praefect Configuration
    praefect['enable'] = true
    praefect['listen_addr'] = '0.0.0.0:2305' # praefect监听端口
    praefect['auto_migrate'] = false
    praefect['database_sslmode'] = 'disable'
    
    
    gitlab_rails['rake_cache_clear'] = false
    gitlab_rails['auto_migrate'] = false
    
    
    
    praefect['auth_token'] = 'PRAEFECT_EXTERNAL_TOKEN' # Praefect集群上的存储库只能由携带此令牌的 Gitaly 客户端访问
    
    praefect['prometheus_listen_addr'] = '0.0.0.0:9652' # 配置普罗米修斯
    
  2. 配置虚拟存储

    PRAEFECT_INTERNAL_TOKEN:Praefect 在与集群中的 Gitaly 节点通信时使用的密码

    praefect['virtual_storages'] = {
      'default' => {
        'nodes' => {
          'gitaly-1' => {
            'address' => 'tcp://10.0.10.32:8075',
            'token'   => 'PRAEFECT_INTERNAL_TOKEN'
          },
          'gitaly-2' => {
            'address' => 'tcp://10.0.10.45:8075',
            'token'   => 'PRAEFECT_INTERNAL_TOKEN'
          },
          'gitaly-3' => {
            'address' => 'tcp://10.0.10.33:8075',
            'token'   => 'PRAEFECT_INTERNAL_TOKEN'
          },
        }
      }
    }
    
  3. 保存配置文件,并重新配置

    gitlab-ctl reconfigure
    

    将praefect[‘auto_migrate’]的值从false改为true

    为确保数据库仅在重新配置期间运行migrate,而不是在升级时,请配置:

    sudo touch /etc/gitlab/skip-auto-reconfigure
    

    然后运行:

    gitlab-ctl reconfigure
    

    请重启Praefect

    gitlab-ctl restart praefect
    
  4. 验证 Praefect 是否可以访问 PostgreSQL

    sudo -u git /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-ping
    

配置Gitaly

  1. 编辑配置文件/etc/gitlab/gitlab.rb

    PRAEFECT_INTERNAL_TOKEN:Praefect 在与集群中的 Gitaly 节点通信时使用的密码,用于 Praefect 集群内的复制流量

    # external_url 'http://gitlab.example.com' # 此配置需要注释
    postgresql['enable'] = false
    redis['enable'] = false
    nginx['enable'] = false
    grafana['enable'] = false
    puma['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
    prometheus_monitoring['enable'] = false
    gitlab_kas['enable'] = false
    
    # Enable only the Gitaly service
    gitaly['enable'] = true
    
    # Enable Prometheus if needed
    prometheus['enable'] = true
    
    # Disable database migrations to prevent database connections during 'gitlab-ctl reconfigure'
    gitlab_rails['auto_migrate'] = false
    
    
    gitaly['listen_addr'] = '0.0.0.0:8075' # 配置gitaly监听端口
    gitaly['prometheus_listen_addr'] = '0.0.0.0:9236'
    gitaly['auth_token'] = 'PRAEFECT_INTERNAL_TOKEN'
    gitlab_rails['internal_api_url'] = 'http://43.156.96.229'
    
  2. 拷贝Praefect节点上的/etc/gitlab/gitlab-secrets.json文件到每个Gitaly节点相同路径上。

    重新配置:

    gitlab-ctl reconfigure
    
  3. 配置internal_api_url,这是git push操作依赖的配置。

    gitlab_rails['internal_api_url'] = 'http://10.0.10.31'
    
  4. 配置git仓库数据库存储位置

    git_data_dirs({
      "gitaly-1" => {
        "path" => "/var/opt/gitlab/git-data"
      },
      "gitaly-2" => {
        "path" => "/var/opt/gitlab/git-data"
      },
      "gitaly-3" => {
        "path" => "/var/opt/gitlab/git-data"
      }
    })
    
  5. 保存配置并重新配置

    gitlab-ctl reconfigure
    
  6. 重启gitaly

    gitlab-ctl restart gitaly
    

必须为每个 Gitaly 节点完成上述步骤!

  1. 检查:

    连接到praefect集群并运行检查程序

    sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes
    

配置Gitlab

  1. 编辑配置文件

    PRAEFECT_EXTERNAL_TOKEN: Praefect 集群上的存储库只能由此密码的 Gitaly 客户端访问

    external_url 'http://10.0.10.31'
    gitaly['enable'] = false
    git_data_dirs({
      "default" => {
        "gitaly_address" => "tcp://10.0.10.35:2305",
        "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN'
      }
    })
    
    prometheus['scrape_configs'] = [
      {
        'job_name' => 'praefect',
        'static_configs' => [
          'targets' => [
            '10.0.10.35:9652', # praefect-1
          ]
        ]
      },
      {
        'job_name' => 'praefect-gitaly',
        'static_configs' => [
          'targets' => [
            '10.0.10.32:9236', # gitaly-1
            '10.0.10.45:9236', # gitaly-2
            '10.0.10.33:9236', # gitaly-3
          ]
        ]
      }
    ]
    
    
  2. 拷贝Praefect节点上的/etc/gitlab/gitlab-secrets.json文件到Gitlab节点相同路径上。

    重新配置:

    gitlab-ctl reconfigure
    
  3. 在每个Gitaly节点上验证Git Hooks对Gitlab可达,在每个Gitaly节点上运行

    • 对于 GitLab 15.3 及更高版本,运行sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml.
    • 对于 GitLab 15.2 及更早版本,运行sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml.
  4. 验证 GitLab 是否可以到达 Praefect:

    gitlab-rake gitlab:gitaly:check
    
  5. 最后登录Gitlab进行验证看看一切是否顺利

重置Gitaly节点

  1. gitlab主节点应该停止写入服务,等待gitaly数据同步完成

  2. 重装该节点的操作系统和Gitlab

  3. 按照配置Gitaly章节重新配置Gitaly

  4. 将其他节点的目录/var/opt/gitlab/git-data/repositories/@hashed进行压缩打包

  5. 拷贝到需要重置的节点进行替换,注意权限为git:git。

  6. 重新启动该gitaly节点

  7. 登录praefect检查数据:

    sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss -partially-replicated
    
    

​ 打印以下内容表示同步成功:

  ```
  Virtual storage: default
    All repositories are up to date!
  ```