[分享] 查找具有correlation ID的相关日志条目

查找具有correlation ID的相关日志条目

GitLab实例记录大多数请求的唯一请求跟踪ID(称为“correlation ID”)。每个到GitLab的单独请求都有自己的correlationID,然后该ID会被记录到每个GitLab组件的请求日志中。这使得跟踪分布式系统中的行为变得更加容易。如果没有这个ID,就很难或不可能匹配相关的日志条目。

识别请求的correlation ID

correlation ID记录在结构化日志中键correlation_id下,并记录在GitLab发送的所有响应报头x-request-id下。您可以通过搜索任一位置来找到correlation ID。

在游览器中获取correlation ID

要定位相关请求并查看correlation ID:

  1. 在网络监视器中启用保留日志,gitlab中的某些操作会在您提交表单后快速重定向,因此开启此选项有助于捕获所有相关活动
  2. 为了帮助隔离您正在寻找的请求,您可以筛选document请求
  3. 选择一个请求
  4. 转到Headers部分并查找Response Headers。找到一个x-request-id标头,它的值是由GitLab为请求随机生成的。

从日志中获取correlation ID

找到正确correlation ID的另一种方法是搜索或监控日志,并查找正在监控的日志条目的correlation_id值。

例如,假设你想知道当你在GitLab中重现一个动作时发生了什么或者破坏了什么。您可以跟踪GitLab日志,过滤到用户的请求,然后观察请求,直到您看到您感兴趣的内容。

从curl中获取correlation ID

如果使用curl,那么可以使用--verbose选项来显示请求和响应头以及其他调试信息。

[root@VM-0-10-centos ~]# curl --verbose  http://<host_ip>/test-group-1/subgroup-1/testx/-/issues
* About to connect() to <host_ip> port 80 (#0)
*   Trying <host_ip>...
* Connected to <host_ip> (<host_ip>) port 80 (#0)
> GET /test-group-1/subgroup-1/testx/-/issues HTTP/1.1
> User-Agent: curl/7.29.0
> Host: <host_ip>
> Accept: */*
>
< HTTP/1.1 302 Found
< Server: nginx
< Date: Wed, 29 Jun 2022 15:22:54
< Content-Type: text/html; charset=utf-8
< Content-Length: 101
< Connection: keep-alive
< Cache-Control: no-cache
< Location: http://<host_ip>/users/sign_in
< Pragma: no-cache
< Set-Cookie: _gitlab_session=d8b79fb34ae33b22663e5b15c159253c; path=/; expires=Wed, 29 Jun 2022 17:22:54; HttpOnly
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: 01G6R0XRWJ9CBFQB8XKX1TFGR9
< X-Runtime: 0.040547
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
< Strict-Transport-Security: max-age=63072000
< Referrer-Policy: strict-origin-when-cross-origin
<
* Connection #0 to host <host_ip> left intact
<html><body>You are being <a href="http://<host_ip>/users/sign_in">redirected</a>.</body></html>[root@VM-0-10-centos ~]#

使用jq

  • 安装jq
https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64
  chmod +x jq-linux64 
  mv jq-linux64 /usr/bin/jq
  • jq教程

  • 使用jq进行过滤

    格式化json输出:

  sudo gitlab-ctl tail gitlab-rails/production_json.log | jq '.'

过滤用户输出:

  sudo gitlab-ctl tail gitlab-rails/production_json.log | jq 'select(.username == "root") | "User: \(.username), \(.method) \(.path), \(.controller)#\(.action), ID: \(.correlation_id)"'

结果:

  "User: root, GET /admin/projects, Admin::ProjectsController#index, ID: 01G6R1CRC1CDDGSMWF8NY47NKC"
  "User: root, GET /-/peek/results, Peek::ResultsController#show, ID: 01G6R1CT424Y39TYEYDS87FEN3"
  "User: root, GET /admin/version_check.json, Admin::VersionCheckController#version_check, ID: 01G6R1CTBN94DRZVFSWP2SSN97"
  "User: root, GET /-/peek/results, Peek::ResultsController#show, ID: 01G6R1CTJ1N8M6NWD01GKNHHZZ"
  "User: root, GET /-/peek/results, Peek::ResultsController#show, ID: 01G6R1CTTNFSVT6K78MT5S016N"
  "User: root, GET /admin/projects/test-group-1/subgroup-2/test-3, Admin::ProjectsController#show, ID: 01G6R1D12Z1MV1VQY2R1KD31W9"
  "User: root, GET /-/peek/results, Peek::ResultsController#show, ID: 01G6R1D1QMVFJ4WVTQYWVC6EHH"
  "User: root, GET /admin/version_check.json, Admin::VersionCheckController#version_check, ID: 01G6R1D1TNCFE69YCSRFCEJMFR"
  "User: root, GET /-/peek/results, Peek::ResultsController#show, ID: 01G6R1D20YQTRAG38F02R3ZEGR"
  "User: root, GET /-/peek/results, Peek::ResultsController#show, ID: 01G6R1D2F4P1G4DSR62A2C033K"
  "User: root, GET /test-group-1/subgroup-2/test-3, ProjectsController#show, ID: 01G6R1D33D3VZC80GXTQF9H3YE"
  "User: root, GET /-/peek/results, Peek::ResultsController#show, ID: 01G6R1D44FVM0F7GB4AF07WBEB"
  "User: root, POST /api/graphql, GraphqlController#execute, ID: 01G6R1D47F0EZ740BWJ9TD3AMS"
  "User: root, POST /api/graphql, GraphqlController#execute, ID: 01G6R1D4CK8EPFR8ZQAACZS3BW"
  "User: root, GET /-/peek/results, Peek::ResultsController#show, ID: 01G6R1D4S1CX2WTMMMFZ6BV7BF"
  "User: root, GET /-/peek/results, Peek::ResultsController#show, ID: 01G6R1D52SGCMV9D2P4X1VMKX4"
  "User: root, GET /test-group-1/subgroup-2/test-3/-/refs/master/logs_tree, Projects::RefsController#logs_tree, ID: 01G6R1D40XW8KG32KQESD64SYG"
  "User: root, GET /-/peek/results, Peek::ResultsController#show, ID: 01G6R1D5M4QM0BERXQRTG99D6K"
  "User: root, GET /test-group-1/subgroup-2/test-3/-/blob/master/README.md, Projects::BlobController#show, ID: 01G6R1D5KVZJSDM5S1N8GSQ03H"
  "User: root, GET /-/peek/results, Peek::ResultsController#show, ID: 01G6R1D5YFS8TCN5YRK7J1QWY7"
  "User: root, POST /api/graphql, GraphqlController#execute, ID: 01G6R1D62S89DSEHSKSSGP4E9E"
  "User: root, GET /-/peek/results, Peek::ResultsController#show, ID: 01G6R1D6FGYQV8B268EMRKXXXK"
  "User: root, GET /test-group-1/subgroup-2/test-3/-/refs/master/logs_tree, Projects::RefsController#logs_tree, ID: 01G6R1D5MABNSCX4ZB3687M64C"
  "User: root, GET /-/peek/results, Peek::ResultsController#show, ID: 01G6R1D6Z0X1Z1XBP1C9FV5AWX"
  "User: root, GET /test-group-1/subgroup-2/test-3/-/refs/master/logs_tree, Projects::RefsController#logs_tree, ID: 01G6R1D6Z6BTY4RHJMD726GT4E"
  "User: root, DELETE /admin/projects/test-group-1/subgroup-2/test-3, Admin::ProjectsController#destroy, ID: 01G6R1DFG146V491B0KSB5GB01"
  "User: root, GET /admin/projects, Admin::ProjectsController#index, ID: 01G6R1DFSRHNE6NN9CHMPRA4S4"
  "User: root, GET /-/peek/results, Peek::ResultsController#show, ID: 01G6R1DGAV5XZ876JHHQ1TZT5Q"
  "User: root, GET /admin/version_check.json, Admin::VersionCheckController#version_check, ID: 01G6R1DGD8D5AEF161EEVAE7XJ"
  "User: root, GET /-/peek/results, Peek::ResultsController#show, ID: 01G6R1DGTDDBAM5MPHEGSQ3DAS"
  "User: root, GET /-/peek/results, Peek::ResultsController#show, ID: 01G6R1DGW9RK393BXXFECGA4N2"
  ......
  • 使用grep
  sudo gitlab-ctl tail gitlab-rails/production_json.log | grep '"username":"root"' | tr ',' '\n' | egrep 'method|path|correlation_id'

结果:

  {"method":"GET"
  "path":"/test-group-1/subgroup-1/testx/-/merge_requests"
  "correlation_id":"01G6R1ZFJ15HH63JZH231NV0JE"
  {"method":"GET"
  "path":"/test-group-1/subgroup-1/testx/-/merge_requests/new"
  "correlation_id":"01G6R1ZJQ96N8EBZPXFP5EW43Y"
  {"method":"GET"
  "path":"/test-group-1/subgroup-1/testx/-/merge_requests/new/branch_from"

在日志中收集correlation ID

有了correlation ID之后,就可以开始搜索相关的日志条目。您可以根据correlation ID本身过滤行。将find和grep组合起来应该足以找到您要查找的条目。

# find <gitlab log directory> -type f -mtime -0 exec grep '<correlation ID>' '{}' '+'
find /var/log/gitlab -type f -mtime 0 -exec grep '01G6SQ2CGDWDK0G2AMY57AFK5G' '{}' '+'
/var/log/gitlab/gitlab-workhorse/current:{"correlation_id":"LOt9hgi1TV4","duration_ms":2478,"host":"gitlab.domain.tld","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.domain.tld/root/linux","remote_addr":"68.0.116.160:0","remote_ip":"[filtered]","status":200,"system":"http","time":"2019-09-17T22:17:19Z","uri":"/root/linux/blob/master/README?format=json\u0026viewer=rich","user_agent":"Mozilla/5.0 (Mac) Gecko Firefox/69.0","written_bytes":1743}
/var/log/gitlab/gitaly/current:{"correlation_id":"LOt9hgi1TV4","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"FindCommits","grpc.request.deadline":"2019-09-17T22:17:47Z","grpc.request.fullMethod":"/gitaly.CommitService/FindCommits","grpc.request.glProjectPath":"root/linux","grpc.request.glRepository":"project-1","grpc.request.repoPath":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"@hashed","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-09-17T22:17:17Z","grpc.time_ms":2319.161,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","span.kind":"server","system":"grpc","time":"2019-09-17T22:17:19Z"}
/var/log/gitlab/gitlab-rails/production_json.log:{"method":"GET","path":"/root/linux/blob/master/README","format":"json","controller":"Projects::BlobController","action":"show","status":200,"duration":2448.77,"view":0.49,"db":21.63,"time":"2019-09-17T22:17:19.800Z","params":[{"key":"viewer","value":"rich"},{"key":"namespace_id","value":"root"},{"key":"project_id","value":"linux"},{"key":"id","value":"master/README"}],"remote_ip":"[filtered]","user_id":2,"username":"bob","ua":"Mozilla/5.0 (Mac) Gecko Firefox/69.0","queue_duration":3.38,"gitaly_calls":1,"gitaly_duration":0.77,"rugged_calls":4,"rugged_duration_ms":28.74,"correlation_id":"LOt9hgi1TV4"}

在高可用架构中搜索correlation ID

如果你已经在你的GitLab基础架构中做了一些水平扩展,那么你将需要搜索你所有的GitLab节点。您可以使用一些日志聚合软件(如Loki、ELK、Splunk或其他)来实现这一点。

您可以使用像Ansible或PSSH(并行SSH)这样的工具,它们可以在服务器间并行执行相同的命令,或者制作自己的解决方案。

1 个赞