[分享] GitLab常用对象迁移和检查

以下介绍在配置了对象存储之后,如何迁移本地的对象到远端OBS,以及如何实时检查本地和对象存储桶的对象个数:

配置

gitlab_rails['artifacts_enabled'] = true
gitlab_rails['artifacts_object_store_enabled'] = true
gitlab_rails['artifacts_object_store_remote_directory'] = "alex-artifacts"
gitlab_rails['artifacts_object_store_connection'] = {
  'provider' => 'AWS',
  'aws_access_key_id' => 'xxxxxx',
  'aws_secret_access_key' => 'xxxxxx',
  'endpoint' => 'https://xxxxxx.cos.ap-shanghai.myqcloud.com',
  'path_style' => 'true'
}

迁移

gitlab-rake gitlab:artifacts:migrate
gitlab-rake "gitlab:lfs:migrate"
gitlab-rake "gitlab:uploads:migrate:all"
gitlab-rake gitlab:external_diffs:force_object_storage
gitlab-rake "gitlab:packages:migrate"
gitlab-rake "gitlab:dependency_proxy:migrate"
gitlab-rake gitlab:terraform_states:migrate
gitlab-rake gitlab:pages:deployments:migrate_to_object_storage

对比本地和对象存储桶中对象个数

gitlab-psql -c "SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts;"
gitlab-psql -c "SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM lfs_objects;"
gitlab-psql -c "SELECT count(*) AS total, sum(case when store = '1' then 1 else 0 end) AS filesystem, sum(case when store = '2' then 1 else 0 end) AS objectstg FROM uploads;"
gitlab-psql -c "SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM packages_package_files;"
gitlab-psql -c "SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM dependency_proxy_blobs;"
gitlab-psql -c "SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM dependency_proxy_manifests;"
gitlab-psql -c "SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM terraform_state_versions;"
gitlab-psql -c "SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM pages_deployments;"

检查本地文件夹的对象文件个数

sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp | wc -l
sudo find /var/opt/gitlab/gitlab-rails/shared/lfs-objects -type f | grep -v tmp | wc -l
sudo find /var/opt/gitlab/gitlab-rails/uploads -type f | grep -v tmp | wc -l
sudo find /var/opt/gitlab/gitlab-rails/external-diffs -type f | grep -v tmp | wc -l
sudo find /var/opt/gitlab/gitlab-rails/shared/dependency_proxy -type f | grep -v tmp | wc -l
sudo find /var/opt/gitlab/gitlab-rails/shared/terraform_state -type f | grep -v tmp | wc -l
sudo find /var/opt/gitlab/gitlab-rails/shared/packages -type f | grep -v tmp | wc -l