Allow to monitor Gerrit on Kubernetes

So far it was only possible to monitor single instance Gerrit servers.
This was due to to the fact that a URL had to be used that pointed to
a dedicated instance, since if multiple replicas would be behind the
instance, the metrics of a random replica would be scraped and not of
all.

Prometheus has a service discovery functionality for deployments running
in Kubernetes. This is now used, when monitoring a Gerrit instance in
Kubernetes. This allows to have a variable number of replicas running,
which will be automatically discovered by Prometheus.

The dashboards were adapted accordingly and allow now to select the
replica to be observed. For now, no summary of all replicas can be
displayed in the dashboards, but that feature is planned to be added
in the future.

Change-Id: I96efc63a192cd90f5e3e91a53dace8e1ae83132e
This commit is contained in:
Thomas Draebing 2020-04-29 17:33:42 +02:00
parent 7663baf7be
commit 451882b7e9
13 changed files with 429 additions and 172 deletions

110
README.md
View file

@ -76,47 +76,75 @@ setup, some configuration is highly dependent on the specific installation.
These options have to be configured in the `./config.yaml` before installing and
are listed here:
| option | description |
|------------------------------------------|------------------------------------------------------------------------------------|
| `gerritServers.[0].host` | Hostname (incl. port, if required) of the Gerrit server to monitor |
| `gerritServers.[0].username` | Username of Gerrit user with 'View Metrics' capabilities |
| `gerritServers.[0].password` | Password of Gerrit user with 'View Metrics' capabilities |
| `gerritServers.[0].promtail.storagePath` | Path to directory, where Promtail is allowed to save files (e.g. `positions.yaml`) |
| `gerritServers.[0].promtail.logPath` | Path to directory containing the Gerrit logs (e.g. `/var/gerrit/logs`) |
| `namespace` | The namespace the charts are installed to |
| `tls.skipVerify` | Whether to skip TLS certificate verification |
| `tls.caCert` | CA certificate used for TLS certificate verification |
| `prometheus.server.host` | Prometheus server ingress hostname |
| `prometheus.server.username` | Username for Prometheus |
| `prometheus.server.password` | Password for Prometheus |
| `prometheus.server.tls.cert` | TLS certificate |
| `prometheus.server.tls.key` | TLS key |
| `prometheus.alertmanager.slack.apiUrl` | API URL of the Slack Webhook |
| `prometheus.alertmanager.slack.channel` | Channel to which the alerts should be posted |
| `loki.host` | Loki ingress hostname |
| `loki.username` | Username for Loki |
| `loki.password` | Password for Loki |
| `loki.s3.protocol` | Protocol used for communicating with S3 |
| `loki.s3.host` | Hostname of the S3 object store |
| `loki.s3.accessToken` | The EC2 accessToken used for authentication with S3 |
| `loki.s3.secret` | The secret associated with the accessToken |
| `loki.s3.bucket` | The name of the S3 bucket |
| `loki.s3.region` | The region in which the S3 bucket is hosted |
| `loki.tls.cert` | TLS certificate |
| `loki.tls.key` | TLS key |
| `grafana.host` | Grafana ingress hostname |
| `grafana.tls.cert` | TLS certificate |
| `grafana.tls.key` | TLS key |
| `grafana.admin.username` | Username for the admin user |
| `grafana.admin.password` | Password for the admin user |
| `grafana.ldap.enabled` | Whether to enable LDAP |
| `grafana.ldap.host` | Hostname of LDAP server |
| `grafana.ldap.port` | Port of LDAP server (Has to be `quoted`!) |
| `grafana.ldap.password` | Password of LDAP server |
| `grafana.ldap.bind_dn` | Bind DN (username) of the LDAP server |
| `grafana.ldap.accountBases` | List of base DNs to discover accounts (Has to have the format `"['a', 'b']"`) |
| `grafana.ldap.groupBases` | List of base DNs to discover groups (Has to have the format `"['a', 'b']"`) |
| `grafana.dashboards.editable` | Whether dashboards can be edited manually in the UI |
| option | description |
|-----------------------------------------|----------------------------------------------------------------------------------------|
| `gerritServers` | List of Gerrit servers to scrape. For details refer to section [below](#gerritServers) |
| `namespace` | The namespace the charts are installed to |
| `tls.skipVerify` | Whether to skip TLS certificate verification |
| `tls.caCert` | CA certificate used for TLS certificate verification |
| `prometheus.server.host` | Prometheus server ingress hostname |
| `prometheus.server.username` | Username for Prometheus |
| `prometheus.server.password` | Password for Prometheus |
| `prometheus.server.tls.cert` | TLS certificate |
| `prometheus.server.tls.key` | TLS key |
| `prometheus.alertmanager.slack.apiUrl` | API URL of the Slack Webhook |
| `prometheus.alertmanager.slack.channel` | Channel to which the alerts should be posted |
| `loki.host` | Loki ingress hostname |
| `loki.username` | Username for Loki |
| `loki.password` | Password for Loki |
| `loki.s3.protocol` | Protocol used for communicating with S3 |
| `loki.s3.host` | Hostname of the S3 object store |
| `loki.s3.accessToken` | The EC2 accessToken used for authentication with S3 |
| `loki.s3.secret` | The secret associated with the accessToken |
| `loki.s3.bucket` | The name of the S3 bucket |
| `loki.s3.region` | The region in which the S3 bucket is hosted |
| `loki.tls.cert` | TLS certificate |
| `loki.tls.key` | TLS key |
| `grafana.host` | Grafana ingress hostname |
| `grafana.tls.cert` | TLS certificate |
| `grafana.tls.key` | TLS key |
| `grafana.admin.username` | Username for the admin user |
| `grafana.admin.password` | Password for the admin user |
| `grafana.ldap.enabled` | Whether to enable LDAP |
| `grafana.ldap.host` | Hostname of LDAP server |
| `grafana.ldap.port` | Port of LDAP server (Has to be `quoted`!) |
| `grafana.ldap.password` | Password of LDAP server |
| `grafana.ldap.bind_dn` | Bind DN (username) of the LDAP server |
| `grafana.ldap.accountBases` | List of base DNs to discover accounts (Has to have the format `"['a', 'b']"`) |
| `grafana.ldap.groupBases` | List of base DNs to discover groups (Has to have the format `"['a', 'b']"`) |
| `grafana.dashboards.editable` | Whether dashboards can be edited manually in the UI |
### `gerritServers`
Two types of Gerrit servers are currently supported, which require different
configuration parameters:
* Kubernetes \
Gerrit installations running in the same Kubernetes cluster as the monitoring
setup. Multiple replicas are supported and automatically discovered.
| option | description |
|----------------------------------------------|----------------------------------------------------------|
| `gerritServers.kubernetes.[*].namespace` | Namespace into which Gerrit was deployed |
| `gerritServers.kubernetes.[*].label.name` | Label name used to select deployments |
| `gerritServers.kubernetes.[*].label.value` | Label value to select deployments |
| `gerritServers.kubernetes.[*].containerName` | Name of container in the pod that runs Gerrit |
| `gerritServers.kubernetes.[*].port` | Container port to be used when scraping |
| `gerritServers.kubernetes.[*].username` | Username of Gerrit user with 'View Metrics' capabilities |
| `gerritServers.kubernetes.[*].password` | Password of Gerrit user with 'View Metrics' capabilities |
* Other \
Gerrit installations with just one replica that can run anywhere, where they
are reachable via HTTP.
| option | description |
|------------------------------------------------|------------------------------------------------------------------------------------|
| `gerritServers.other.[*].host` | Hostname (incl. port, if required) of the Gerrit server to monitor |
| `gerritServers.other.[*].username` | Username of Gerrit user with 'View Metrics' capabilities |
| `gerritServers.other.[*].password` | Password of Gerrit user with 'View Metrics' capabilities |
| `gerritServers.other.[*].promtail.storagePath` | Path to directory, where Promtail is allowed to save files (e.g. `positions.yaml`) |
| `gerritServers.other.[*].promtail.logPath` | Path to directory containing the Gerrit logs (e.g. `/var/gerrit/logs`) |
### Encryption

View file

@ -7,7 +7,11 @@ metadata:
namespace: #@ data.values.namespace
data:
#@yaml/text-templated-strings
#@ for gerrit in data.values.gerritServers:
#@ for gerrit in data.values.gerritServers.kubernetes:
.pwd_(@= gerrit.namespace @): #@ base64.encode(gerrit.password)
#@ end
#@yaml/text-templated-strings
#@ for gerrit in data.values.gerritServers.other:
.pwd_(@= gerrit.host @): #@ base64.encode(gerrit.password)
#@ end

View file

@ -1127,7 +1127,43 @@ serverFiles:
- /etc/config/alerts
scrape_configs:
#@ for gerrit in data.values.gerritServers:
#@ for gerrit in data.values.gerritServers.kubernetes:
- job_name: #@ "gerrit-{}".format(gerrit.namespace)
metrics_path: /a/plugins/metrics-reporter-prometheus/metrics
scheme: http
basic_auth:
username: #@ gerrit.username
password_file: #@ "/etc/secrets/.pwd_{}".format(gerrit.namespace)
kubernetes_sd_configs:
- role: pod
namespaces:
names:
- #@ gerrit.namespace
relabel_configs:
- source_labels:
- #@ "__meta_kubernetes_pod_label_{}".format(gerrit.label.name)
regex: #@ gerrit.label.value
action: keep
- source_labels: [__meta_kubernetes_pod_container_name]
regex: #@ gerrit.containerName
action: keep
- source_labels: [__meta_kubernetes_pod_container_port_number]
regex: #@ gerrit.port
action: keep
- source_labels: [__meta_kubernetes_namespace]
regex: (.*)
target_label: instance
replacement: kubernetes_$1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
regex: (.*)
target_label: replica
replacement: $1
action: replace
#@ end
#@ for gerrit in data.values.gerritServers.other:
- job_name: #@ "gerrit-{}".format(gerrit.host)
metrics_path: /a/plugins/metrics-reporter-prometheus/metrics
scheme: https
@ -1141,6 +1177,8 @@ serverFiles:
static_configs:
- targets:
- #@ gerrit.host
labels:
replica: main
basic_auth:
username: #@ gerrit.username
password_file: #@ "/etc/secrets/.pwd_{}".format(gerrit.host)

View file

@ -1,4 +1,14 @@
gerritServers:
kubernetes:
- namespace: default
label:
name: app
value: gerrit-slave
containerName: gerrit-slave
port: 8080
username: admin
password: secret
other:
- host: gerrit.example.com
username: admin
password: secret

View file

@ -62,22 +62,22 @@
"steppedLine": false,
"targets": [
{
"expr": "100-caches_memory_hit_ratio_accounts{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_accounts{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "accounts",
"refId": "A"
},
{
"expr": "100-caches_memory_hit_ratio_groups{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_groups{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "groups",
"refId": "B"
},
{
"expr": "100-caches_memory_hit_ratio_groups_byuuid{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_groups_byuuid{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "groups_byuuid",
"refId": "C"
},
{
"expr": "100-caches_memory_hit_ratio_ldap_groups_byinclude{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_ldap_groups_byinclude{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "ldap_groups_byinclude",
"refId": "D"
}
@ -169,12 +169,12 @@
"steppedLine": false,
"targets": [
{
"expr": "100-caches_memory_hit_ratio_conflicts{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_conflicts{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "conflicts",
"refId": "A"
},
{
"expr": "100-caches_memory_hit_ratio_mergeability{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_mergeability{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "mergeability",
"refId": "B"
}
@ -265,17 +265,17 @@
"steppedLine": false,
"targets": [
{
"expr": "100-caches_memory_hit_ratio_change_kind{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_change_kind{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "change_kind",
"refId": "B"
},
{
"expr": "100-caches_memory_hit_ratio_change_notes{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_change_notes{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "change_notes",
"refId": "C"
},
{
"expr": "100-caches_memory_hit_ratio_changeid_project{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_changeid_project{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "changeid_project",
"refId": "D"
}
@ -365,12 +365,12 @@
"steppedLine": false,
"targets": [
{
"expr": "100-caches_memory_hit_ratio_project_list{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_project_list{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "project list",
"refId": "A"
},
{
"expr": "100-caches_memory_hit_ratio_projects{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_projects{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "projects",
"refId": "B"
}
@ -461,17 +461,17 @@
"steppedLine": false,
"targets": [
{
"expr": "100-caches_memory_hit_ratio_diff{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_diff{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "diff",
"refId": "A"
},
{
"expr": "100-caches_memory_hit_ratio_diff_intraline{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_diff_intraline{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "diff intraline",
"refId": "B"
},
{
"expr": "100-caches_memory_hit_ratio_diff_summary{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_diff_summary{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "diff summary",
"refId": "C"
}
@ -563,22 +563,22 @@
"steppedLine": false,
"targets": [
{
"expr": "100-caches_memory_hit_ratio_web_sessions{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_web_sessions{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "web sessions",
"refId": "A"
},
{
"expr": "100-caches_memory_hit_ratio_sshkeys{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_sshkeys{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "sshkeys",
"refId": "B"
},
{
"expr": "100-caches_memory_hit_ratio_git_tags{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_git_tags{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "git tags",
"refId": "D"
},
{
"expr": "100-caches_memory_hit_ratio_permission_sort{instance=\"$instance\"}",
"expr": "100-caches_memory_hit_ratio_permission_sort{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "permission_sort",
"refId": "C"
}
@ -655,6 +655,31 @@
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"text": "",
"value": ""
},
"datasource": "Prometheus",
"definition": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
"hide": 0,
"includeAll": false,
"label": "Replica",
"multi": false,
"name": "replica",
"options": [],
"query": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},

View file

@ -60,7 +60,7 @@
"steppedLine": false,
"targets": [
{
"expr": "increase(git_upload_pack_request_count_CLONE_total{instance=\"$instance\"}[2m])/2",
"expr": "increase(git_upload_pack_request_count_CLONE_total{instance=\"$instance\",replica=\"$replica\"}[2m])/2",
"legendFormat": "clone",
"refId": "A"
}
@ -149,7 +149,7 @@
"steppedLine": false,
"targets": [
{
"expr": "increase(git_upload_pack_request_count_FETCH_total{instance=\"$instance\"}[2m])/2",
"expr": "increase(git_upload_pack_request_count_FETCH_total{instance=\"$instance\",replica=\"$replica\"}[2m])/2",
"legendFormat": "fetch",
"refId": "A"
}
@ -253,7 +253,7 @@
"steppedLine": false,
"targets": [
{
"expr": "git_upload_pack_pack_bytes_CLONE{instance=\"$instance\"}",
"expr": "git_upload_pack_pack_bytes_CLONE{instance=\"$instance\",replica=\"$replica\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "quantile: {{quantile}}",
@ -360,7 +360,7 @@
"steppedLine": false,
"targets": [
{
"expr": "git_upload_pack_pack_bytes_FETCH{instance=\"$instance\"}",
"expr": "git_upload_pack_pack_bytes_FETCH{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
}
@ -438,6 +438,31 @@
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"text": "",
"value": ""
},
"datasource": "Prometheus",
"definition": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
"hide": 0,
"includeAll": false,
"label": "Replica",
"multi": false,
"name": "replica",
"options": [],
"query": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},

View file

@ -73,7 +73,7 @@
"steppedLine": false,
"targets": [
{
"expr": "receivecommits_latency_total{instance=\"$instance\"}",
"expr": "receivecommits_latency_total{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@ -174,7 +174,7 @@
"steppedLine": false,
"targets": [
{
"expr": "http_server_rest_api_server_latency_total{instance=\"$instance\"}",
"expr": "http_server_rest_api_server_latency_total{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@ -275,7 +275,7 @@
"steppedLine": false,
"targets": [
{
"expr": "query_query_latency_total{instance=\"$instance\"}",
"expr": "query_query_latency_total{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@ -376,7 +376,7 @@
"steppedLine": false,
"targets": [
{
"expr": "http_server_rest_api_server_latency_restapi_change_GetDetail{instance=\"$instance\"}",
"expr": "http_server_rest_api_server_latency_restapi_change_GetDetail{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@ -477,7 +477,7 @@
"steppedLine": false,
"targets": [
{
"expr": "query_query_latency_changes{instance=\"$instance\"}",
"expr": "query_query_latency_changes{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@ -578,7 +578,7 @@
"steppedLine": false,
"targets": [
{
"expr": "http_server_rest_api_server_latency_restapi_change_GetDiff{instance=\"$instance\"}",
"expr": "http_server_rest_api_server_latency_restapi_change_GetDiff{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@ -679,7 +679,7 @@
"steppedLine": false,
"targets": [
{
"expr": "http_server_rest_api_server_latency_restapi_change_ListChangeComments{instance=\"$instance\"}",
"expr": "http_server_rest_api_server_latency_restapi_change_ListChangeComments{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@ -780,7 +780,7 @@
"steppedLine": false,
"targets": [
{
"expr": "http_server_rest_api_server_latency_restapi_change_GetChange{instance=\"$instance\"}",
"expr": "http_server_rest_api_server_latency_restapi_change_GetChange{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@ -881,7 +881,7 @@
"steppedLine": false,
"targets": [
{
"expr": "http_server_rest_api_server_latency_restapi_change_ListChangeRobotComments{instance=\"$instance\"}",
"expr": "http_server_rest_api_server_latency_restapi_change_ListChangeRobotComments{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@ -983,7 +983,7 @@
"steppedLine": false,
"targets": [
{
"expr": "http_server_rest_api_server_latency_restapi_change_GetCommit{instance=\"$instance\"}",
"expr": "http_server_rest_api_server_latency_restapi_change_GetCommit{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@ -1084,7 +1084,7 @@
"steppedLine": false,
"targets": [
{
"expr": "http_server_rest_api_server_latency_restapi_change_PostReview{instance=\"$instance\"}",
"expr": "http_server_rest_api_server_latency_restapi_change_PostReview{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@ -1185,7 +1185,7 @@
"steppedLine": false,
"targets": [
{
"expr": "http_server_rest_api_server_latency_restapi_change_GetRevisionActions{instance=\"$instance\"}",
"expr": "http_server_rest_api_server_latency_restapi_change_GetRevisionActions{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@ -1263,6 +1263,31 @@
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"text": "",
"value": ""
},
"datasource": "Prometheus",
"definition": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
"hide": 0,
"includeAll": false,
"label": "Replica",
"multi": false,
"name": "replica",
"options": [],
"query": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},
@ -1291,4 +1316,4 @@
"list": []
},
"version": 1
}
}

View file

@ -143,7 +143,7 @@
],
"targets": [
{
"expr": "max(gerrit_build_info{instance=\"$instance\"}) by (instance, version, revision, javaversion)",
"expr": "max(gerrit_build_info{instance=\"$instance\",replica=\"$replica\"}) by (instance, version, revision, javaversion)",
"format": "table",
"instant": true,
"interval": "",
@ -232,7 +232,7 @@
"pluginVersion": "6.7.1",
"targets": [
{
"expr": "(rate(proc_cpu_usage{instance=\"$instance\"}[5m])/proc_cpu_num_cores{instance=\"$instance\"})*100",
"expr": "(rate(proc_cpu_usage{instance=\"$instance\",replica=\"$replica\"}[5m])/proc_cpu_num_cores{instance=\"$instance\",replica=\"$replica\"})*100",
"refId": "A"
}
],
@ -302,7 +302,7 @@
"pluginVersion": "6.7.1",
"targets": [
{
"expr": "(proc_jvm_memory_heap_used{instance=\"$instance\"}/proc_jvm_memory_heap_committed{instance=\"$instance\"})*100",
"expr": "(proc_jvm_memory_heap_used{instance=\"$instance\",replica=\"$replica\"}/proc_jvm_memory_heap_committed{instance=\"$instance\",replica=\"$replica\"})*100",
"refId": "A"
}
],
@ -352,17 +352,17 @@
"pluginVersion": "6.7.1",
"targets": [
{
"expr": "proc_jvm_thread_num_daemon_live{instance=\"$instance\"}",
"expr": "proc_jvm_thread_num_daemon_live{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "current live threads (daemon)",
"refId": "A"
},
{
"expr": "proc_jvm_thread_num_live{instance=\"$instance\"}",
"expr": "proc_jvm_thread_num_live{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "current live threads",
"refId": "B"
},
{
"expr": "proc_jvm_thread_num_peak_live{instance=\"$instance\"}",
"expr": "proc_jvm_thread_num_peak_live{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "peak of live threads",
"refId": "C"
}
@ -420,7 +420,7 @@
"pluginVersion": "6.7.1",
"targets": [
{
"expr": "http_server_rest_api_server_latency_total{quantile=\"0.99\", instance=\"$instance\"}",
"expr": "http_server_rest_api_server_latency_total{quantile=\"0.99\", instance=\"$instance\",replica=\"$replica\"}",
"intervalFactor": 4,
"legendFormat": "quantile {{quantile}}",
"refId": "A"
@ -493,7 +493,7 @@
"pluginVersion": "6.7.1",
"targets": [
{
"expr": "(increase(http_server_error_count_total_total{instance=\"$instance\"}[5m]) - increase(http_server_error_count_404_total{instance=\"$instance\"}[5m]) - increase(http_server_error_count_401_total{instance=\"$instance\"}[5m])) / (increase(http_server_success_count_total_total{instance=\"$instance\"}[5m]) + increase(http_server_error_count_total_total{instance=\"$instance\"}[5m]) - increase(http_server_error_count_404_total{instance=\"$instance\"}[5m]) - increase(http_server_error_count_401_total{instance=\"$instance\"}[5m]))*100",
"expr": "(increase(http_server_error_count_total_total{instance=\"$instance\",replica=\"$replica\"}[5m]) - increase(http_server_error_count_404_total{instance=\"$instance\",replica=\"$replica\"}[5m]) - increase(http_server_error_count_401_total{instance=\"$instance\",replica=\"$replica\"}[5m])) / (increase(http_server_success_count_total_total{instance=\"$instance\",replica=\"$replica\"}[5m]) + increase(http_server_error_count_total_total{instance=\"$instance\",replica=\"$replica\"}[5m]) - increase(http_server_error_count_404_total{instance=\"$instance\",replica=\"$replica\"}[5m]) - increase(http_server_error_count_401_total{instance=\"$instance\",replica=\"$replica\"}[5m]))*100",
"refId": "A"
}
],
@ -557,7 +557,7 @@
"steppedLine": true,
"targets": [
{
"expr": "up{instance=\"$instance\"}",
"expr": "up{instance=\"$instance\",replica=\"$replica\"}",
"format": "time_series",
"legendFormat": "{{instance}}",
"refId": "A"
@ -669,7 +669,7 @@
"tableColumn": "",
"targets": [
{
"expr": "avg_over_time(up{instance=\"$instance\"}[1d])*100",
"expr": "avg_over_time(up{instance=\"$instance\",replica=\"$replica\"}[1d])*100",
"format": "time_series",
"legendFormat": "{{instance}}",
"refId": "A"
@ -798,98 +798,98 @@
"steppedLine": false,
"targets": [
{
"expr": "(increase(http_server_success_count_200_total{instance=\"$instance\"}[5m]))/5",
"expr": "(increase(http_server_success_count_200_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "200",
"refId": "A"
},
{
"expr": "(increase(http_server_success_count_201_total{instance=\"$instance\"}[5m]))/5",
"expr": "(increase(http_server_success_count_201_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "201",
"refId": "C"
},
{
"expr": "(increase(http_server_success_count_204_total{instance=\"$instance\"}[5m]))/5",
"expr": "(increase(http_server_success_count_204_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "204",
"refId": "D"
},
{
"expr": "(increase(http_server_success_count_301_total{instance=\"$instance\"}[5m]))/5",
"expr": "(increase(http_server_success_count_301_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "301",
"refId": "E"
},
{
"expr": "(increase(http_server_success_count_304_total{instance=\"$instance\"}[5m]))/5",
"expr": "(increase(http_server_success_count_304_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "304",
"refId": "F"
},
{
"expr": "(increase(http_server_error_count_400_total{instance=\"$instance\"}[5m]))/5",
"expr": "(increase(http_server_error_count_400_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "400",
"refId": "B"
},
{
"expr": "(increase(http_server_error_count_401_total{instance=\"$instance\"}[5m]))/5",
"expr": "(increase(http_server_error_count_401_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "401",
"refId": "G"
},
{
"expr": "(increase(http_server_error_count_403_total{instance=\"$instance\"}[5m]))/5",
"expr": "(increase(http_server_error_count_403_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "403",
"refId": "H"
},
{
"expr": "(increase(http_server_error_count_404_total{instance=\"$instance\"}[5m]))/5",
"expr": "(increase(http_server_error_count_404_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"instant": false,
"intervalFactor": 1,
"legendFormat": "404",
"refId": "I"
},
{
"expr": "(increase(http_server_error_count_405_total{instance=\"$instance\"}[5m]))/5",
"expr": "(increase(http_server_error_count_405_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "405",
"refId": "J"
},
{
"expr": "(increase(http_server_error_count_409_total{instance=\"$instance\"}[5m]))/5",
"expr": "(increase(http_server_error_count_409_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "409",
"refId": "K"
},
{
"expr": "(increase(http_server_error_count_412_total{instance=\"$instance\"}[5m]))/5",
"expr": "(increase(http_server_error_count_412_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "412",
"refId": "L"
},
{
"expr": "(increase(http_server_error_count_422_total{instance=\"$instance\"}[5m]))/5",
"expr": "(increase(http_server_error_count_422_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "422",
"refId": "M"
},
{
"expr": "(increase(http_server_error_count_500_total{instance=\"$instance\"}[5m]))/5",
"expr": "(increase(http_server_error_count_500_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "500",
"refId": "N"
},
{
"expr": "(increase(http_server_error_count_501_total{instance=\"$instance\"}[5m]))/5",
"expr": "(increase(http_server_error_count_501_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "501",
"refId": "O"
},
{
"expr": "(increase(http_server_error_count_503_total{instance=\"$instance\"}[5m]))/5",
"expr": "(increase(http_server_error_count_503_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "503",
"refId": "P"
@ -981,73 +981,73 @@
"steppedLine": false,
"targets": [
{
"expr": "increase(events_assignee_changed_total{instance=\"$instance\"}[5m])",
"expr": "increase(events_assignee_changed_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "assignee changed",
"refId": "A"
},
{
"expr": "increase(events_change_abandoned_total{instance=\"$instance\"}[5m])",
"expr": "increase(events_change_abandoned_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "change abandoned",
"refId": "B"
},
{
"expr": "increase(events_change_merged_total{instance=\"$instance\"}[5m])",
"expr": "increase(events_change_merged_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "change merged",
"refId": "C"
},
{
"expr": "increase(events_comment_added_total{instance=\"$instance\"}[5m])",
"expr": "increase(events_comment_added_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "comment added",
"refId": "D"
},
{
"expr": "increase(events_patchset_created_total{instance=\"$instance\"}[5m])",
"expr": "increase(events_patchset_created_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "patchset created",
"refId": "E"
},
{
"expr": "increase(events_ref_replicated_total{instance=\"$instance\"}[5m])",
"expr": "increase(events_ref_replicated_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "ref replicated",
"refId": "F"
},
{
"expr": "increase(events_ref_updated_total{instance=\"$instance\"}[5m])",
"expr": "increase(events_ref_updated_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "ref updated",
"refId": "G"
},
{
"expr": "increase(events_reviewer_added_total{instance=\"$instance\"}[5m])",
"expr": "increase(events_reviewer_added_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "reviewer added",
"refId": "H"
},
{
"expr": "increase(events_reviewer_deleted_total{instance=\"$instance\"}[5m])",
"expr": "increase(events_reviewer_deleted_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "reviewer deleted",
"refId": "I"
},
{
"expr": "increase(events_topic_changed_total{instance=\"$instance\"}[5m])",
"expr": "increase(events_topic_changed_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "topic changed",
"refId": "J"
},
{
"expr": "increase(events_vote_deleted_total{instance=\"$instance\"}[5m])",
"expr": "increase(events_vote_deleted_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "vote deleted",
"refId": "K"
},
{
"expr": "increase(events_wip_state_changed_total{instance=\"$instance\"}[5m])",
"expr": "increase(events_wip_state_changed_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "wip state changed",
"refId": "L"
@ -1140,13 +1140,13 @@
"steppedLine": false,
"targets": [
{
"expr": "increase(git_upload_pack_request_count_FETCH_total{instance=\"$instance\"}[5m])",
"expr": "increase(git_upload_pack_request_count_FETCH_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 4,
"legendFormat": "Fetch",
"refId": "B"
},
{
"expr": "increase(git_upload_pack_request_count_CLONE_total{instance=\"$instance\"}[5m])",
"expr": "increase(git_upload_pack_request_count_CLONE_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 4,
"legendFormat": "Clone",
"refId": "A"
@ -1227,6 +1227,31 @@
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"text": "",
"value": ""
},
"datasource": "Prometheus",
"definition": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
"hide": 0,
"includeAll": false,
"label": "Replica",
"multi": false,
"name": "replica",
"options": [],
"query": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},

View file

@ -59,7 +59,7 @@
"steppedLine": false,
"targets": [
{
"expr": "proc_cpu_system_load{instance=\"$instance\"}",
"expr": "proc_cpu_system_load{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "system load",
"refId": "A"
}
@ -148,23 +148,23 @@
"steppedLine": false,
"targets": [
{
"expr": "proc_jvm_memory_heap_committed{instance=\"$instance\"}",
"expr": "proc_jvm_memory_heap_committed{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "committed heap",
"refId": "C"
},
{
"expr": "proc_jvm_memory_heap_used{instance=\"$instance\"}",
"expr": "proc_jvm_memory_heap_used{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "used heap",
"refId": "B"
},
{
"expr": "jgit_block_cache_cache_used{instance=\"$instance\"}",
"expr": "jgit_block_cache_cache_used{instance=\"$instance\",replica=\"$replica\"}",
"instant": false,
"legendFormat": "JGit block cache",
"refId": "A"
},
{
"expr": "proc_jvm_memory_non_heap_used{instance=\"$instance\"}",
"expr": "proc_jvm_memory_non_heap_used{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "used non-heap",
"refId": "D"
}
@ -261,12 +261,12 @@
"steppedLine": false,
"targets": [
{
"expr": "rate(proc_cpu_usage{instance=\"$instance\"}[5m])",
"expr": "rate(proc_cpu_usage{instance=\"$instance\",replica=\"$replica\"}[5m])",
"legendFormat": "used CPUs",
"refId": "A"
},
{
"expr": "proc_cpu_num_cores{instance=\"$instance\"}",
"expr": "proc_cpu_num_cores{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "available CPUs",
"refId": "B"
}
@ -365,12 +365,12 @@
"steppedLine": false,
"targets": [
{
"expr": "increase(proc_jvm_gc_time_G1_Young_Generation{instance=\"$instance\"}[2m])/increase(proc_uptime{instance=\"$instance\"}[2m])",
"expr": "increase(proc_jvm_gc_time_G1_Young_Generation{instance=\"$instance\",replica=\"$replica\"}[2m])/increase(proc_uptime{instance=\"$instance\",replica=\"$replica\"}[2m])",
"legendFormat": "gc time G1 young gen",
"refId": "B"
},
{
"expr": "increase(proc_jvm_gc_time_G1_Old_Generation{instance=\"$instance\"}[2m])/increase(proc_uptime{instance=\"$instance\"}[2m])",
"expr": "increase(proc_jvm_gc_time_G1_Old_Generation{instance=\"$instance\",replica=\"$replica\"}[2m])/increase(proc_uptime{instance=\"$instance\",replica=\"$replica\"}[2m])",
"legendFormat": "gc time G1 old gen",
"refId": "A"
}
@ -459,7 +459,7 @@
"steppedLine": false,
"targets": [
{
"expr": "proc_jvm_thread_num_live{instance=\"$instance\"}",
"expr": "proc_jvm_thread_num_live{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "Java live threads",
"refId": "A"
}
@ -548,12 +548,12 @@
"steppedLine": false,
"targets": [
{
"expr": "jgit_block_cache_open_files{instance=\"$instance\"}",
"expr": "jgit_block_cache_open_files{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "jgit block cache",
"refId": "B"
},
{
"expr": "proc_num_open_fds{instance=\"$instance\"}-jgit_block_cache_open_files{instance=\"$instance\"}",
"expr": "proc_num_open_fds{instance=\"$instance\",replica=\"$replica\"}-jgit_block_cache_open_files{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "other",
"refId": "C"
}
@ -651,14 +651,14 @@
"steppedLine": false,
"targets": [
{
"expr": "increase(jgit_block_cache_miss_count{instance=\"$instance\"}[2m])/(increase(jgit_block_cache_hit_count{instance=\"$instance\"}[2m])+increase(jgit_block_cache_miss_count{instance=\"$instance\"}[2m]))",
"expr": "increase(jgit_block_cache_miss_count{instance=\"$instance\",replica=\"$replica\"}[2m])/(increase(jgit_block_cache_hit_count{instance=\"$instance\",replica=\"$replica\"}[2m])+increase(jgit_block_cache_miss_count{instance=\"$instance\",replica=\"$replica\"}[2m]))",
"format": "time_series",
"instant": false,
"legendFormat": "miss ratio",
"refId": "A"
},
{
"expr": "increase(jgit_block_cache_eviction_count{instance=\"$instance\"}[2m])/(increase(jgit_block_cache_hit_count{instance=\"$instance\"}[2m])+increase(jgit_block_cache_miss_count{instance=\"$instance\"}[2m]))",
"expr": "increase(jgit_block_cache_eviction_count{instance=\"$instance\",replica=\"$replica\"}[2m])/(increase(jgit_block_cache_hit_count{instance=\"$instance\",replica=\"$replica\"}[2m])+increase(jgit_block_cache_miss_count{instance=\"$instance\",replica=\"$replica\"}[2m]))",
"format": "time_series",
"instant": false,
"legendFormat": "eviction ratio",
@ -739,6 +739,31 @@
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"text": "",
"value": ""
},
"datasource": "Prometheus",
"definition": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
"hide": 0,
"includeAll": false,
"label": "Replica",
"multi": false,
"name": "replica",
"options": [],
"query": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},

View file

@ -83,22 +83,22 @@
"steppedLine": false,
"targets": [
{
"expr": "queue_ssh_batch_worker_active_threads{instance=\"$instance\"}",
"expr": "queue_ssh_batch_worker_active_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "batch threads",
"refId": "C"
},
{
"expr": "queue_ssh_batch_worker_pool_size{instance=\"$instance\"}",
"expr": "queue_ssh_batch_worker_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "batch pool size",
"refId": "D"
},
{
"expr": "queue_ssh_interactive_worker_active_threads{instance=\"$instance\"}",
"expr": "queue_ssh_interactive_worker_active_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "interactive threads",
"refId": "A"
},
{
"expr": "queue_ssh_interactive_worker_pool_size{instance=\"$instance\"}",
"expr": "queue_ssh_interactive_worker_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "interactive pool size",
"refId": "B"
}
@ -208,22 +208,22 @@
"steppedLine": false,
"targets": [
{
"expr": "http_server_jetty_threadpool_active_threads{instance=\"$instance\"}",
"expr": "http_server_jetty_threadpool_active_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "active threads",
"refId": "A"
},
{
"expr": "http_server_jetty_threadpool_reserved_threads{instance=\"$instance\"}",
"expr": "http_server_jetty_threadpool_reserved_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "reserved threads",
"refId": "C"
},
{
"expr": "http_server_jetty_threadpool_max_pool_size{instance=\"$instance\"}",
"expr": "http_server_jetty_threadpool_max_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "max pool size",
"refId": "B"
},
{
"expr": "http_server_jetty_threadpool_pool_size{instance=\"$instance\"}",
"expr": "http_server_jetty_threadpool_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "pool size",
"refId": "D"
}
@ -333,22 +333,22 @@
"steppedLine": false,
"targets": [
{
"expr": "queue_index_batch_active_threads{instance=\"$instance\"}",
"expr": "queue_index_batch_active_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "batch threads",
"refId": "A"
},
{
"expr": "queue_index_batch_pool_size{instance=\"$instance\"}",
"expr": "queue_index_batch_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "batch pool size",
"refId": "B"
},
{
"expr": "queue_index_interactive_active_threads{instance=\"$instance\"}",
"expr": "queue_index_interactive_active_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "interactive threads",
"refId": "C"
},
{
"expr": "queue_index_interactive_pool_size{instance=\"$instance\"}",
"expr": "queue_index_interactive_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "interactive pool size",
"refId": "D"
}
@ -448,12 +448,12 @@
"steppedLine": false,
"targets": [
{
"expr": "queue_ssh_batch_worker_scheduled_tasks{instance=\"$instance\"}",
"expr": "queue_ssh_batch_worker_scheduled_tasks{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "batch",
"refId": "C"
},
{
"expr": "queue_ssh_interactive_worker_scheduled_tasks{instance=\"$instance\"}",
"expr": "queue_ssh_interactive_worker_scheduled_tasks{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "interactive",
"refId": "A"
}
@ -547,7 +547,7 @@
"steppedLine": false,
"targets": [
{
"expr": "http_server_jetty_threadpool_queue_size{instance=\"$instance\"}",
"expr": "http_server_jetty_threadpool_queue_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "http",
"refId": "A"
}
@ -648,12 +648,12 @@
"steppedLine": false,
"targets": [
{
"expr": "queue_index_batch_scheduled_tasks{instance=\"$instance\"}",
"expr": "queue_index_batch_scheduled_tasks{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "batch",
"refId": "C"
},
{
"expr": "queue_index_interactive_scheduled_tasks{instance=\"$instance\"}",
"expr": "queue_index_interactive_scheduled_tasks{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "interactive",
"refId": "A"
}
@ -771,32 +771,32 @@
"steppedLine": false,
"targets": [
{
"expr": "queue_ssh_stream_worker_active_threads{instance=\"$instance\"}",
"expr": "queue_ssh_stream_worker_active_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "stream",
"refId": "A"
},
{
"expr": "queue_ssh_stream_worker_pool_size{instance=\"$instance\"}",
"expr": "queue_ssh_stream_worker_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "stream pool",
"refId": "B"
},
{
"expr": "queue_send_email_active_threads{instance=\"$instance\"}",
"expr": "queue_send_email_active_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "email",
"refId": "C"
},
{
"expr": "queue_send_email_pool_size{instance=\"$instance\"}",
"expr": "queue_send_email_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "email pool",
"refId": "D"
},
{
"expr": "queue_receive_commits_active_threads{instance=\"$instance\"}",
"expr": "queue_receive_commits_active_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "receive-commit",
"refId": "E"
},
{
"expr": "queue_receive_commits_pool_size{instance=\"$instance\"}",
"expr": "queue_receive_commits_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "receive-commit pool",
"refId": "F"
}
@ -887,7 +887,7 @@
"steppedLine": false,
"targets": [
{
"expr": "rate(http_server_rest_api_count_total_total{instance=\"$instance\"}[5m])",
"expr": "rate(http_server_rest_api_count_total_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"legendFormat": "REST API request rate",
"refId": "A"
}
@ -992,17 +992,17 @@
"steppedLine": false,
"targets": [
{
"expr": "queue_ssh_stream_worker_scheduled_tasks{instance=\"$instance\"}",
"expr": "queue_ssh_stream_worker_scheduled_tasks{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "stream",
"refId": "E"
},
{
"expr": "queue_send_email_scheduled_tasks{instance=\"$instance\"}",
"expr": "queue_send_email_scheduled_tasks{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "email",
"refId": "C"
},
{
"expr": "queue_receive_commits_scheduled_tasks{instance=\"$instance\"}",
"expr": "queue_receive_commits_scheduled_tasks{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "receive-commit",
"refId": "D"
}
@ -1079,6 +1079,31 @@
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"text": "",
"value": ""
},
"datasource": "Prometheus",
"definition": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
"hide": 0,
"includeAll": false,
"label": "Replica",
"multi": false,
"name": "replica",
"options": [],
"query": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},

View file

@ -60,7 +60,7 @@
"steppedLine": false,
"targets": [
{
"expr": "plugins_replication_replication_delay_$target{instance=\"$instance\"}",
"expr": "plugins_replication_replication_delay_$target{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "quantile: {{quantile}}",
"refId": "B"
}
@ -148,7 +148,7 @@
"steppedLine": false,
"targets": [
{
"expr": "plugins_replication_replication_latency_$target{instance=\"$instance\"}",
"expr": "plugins_replication_replication_latency_$target{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
}
@ -225,6 +225,31 @@
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"text": "",
"value": ""
},
"datasource": "Prometheus",
"definition": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
"hide": 0,
"includeAll": false,
"label": "Replica",
"multi": false,
"name": "replica",
"options": [],
"query": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {

View file

@ -1,20 +1,20 @@
#@ load("@ytt:data", "data")
#@ for i in range(len(data.values.gerritServers)):
#@ for i in range(len(data.values.gerritServers.other)):
---
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: #@ "{}/positions.yaml".format(data.values.gerritServers[i].promtail.storagePath)
filename: #@ "{}/positions.yaml".format(data.values.gerritServers.other[i].promtail.storagePath)
clients:
- url: #@ "https://{}/loki/api/v1/push".format(data.values.loki.host)
tls_config:
insecure_skip_verify: #@ data.values.tls.skipVerify
#@ if not data.values.tls.skipVerify:
ca_file: #@ "{}/promtail.ca.crt".format(data.values.gerritServers[i].promtail.storagePath)
ca_file: #@ "{}/promtail.ca.crt".format(data.values.gerritServers.other[i].promtail.storagePath)
#@ end
basic_auth:
username: #@ data.values.loki.username
@ -26,8 +26,8 @@ scrape_configs:
- localhost
labels:
job: gerrit_error
__path__: #@ "{}/error_log.json".format(data.values.gerritServers[i].promtail.logPath)
host: #@ data.values.gerritServers[i].host
__path__: #@ "{}/error_log.json".format(data.values.gerritServers.other[i].promtail.logPath)
host: #@ data.values.gerritServers.other[i].host
entry_parser: raw
pipeline_stages:
- json:
@ -57,8 +57,8 @@ scrape_configs:
- localhost
labels:
job: gerrit_httpd
__path__: #@ "{}/httpd_log.json".format(data.values.gerritServers[i].promtail.logPath)
host: #@ data.values.gerritServers[i].host
__path__: #@ "{}/httpd_log.json".format(data.values.gerritServers.other[i].promtail.logPath)
host: #@ data.values.gerritServers.other[i].host
entry_parser: raw
pipeline_stages:
- json:
@ -76,8 +76,8 @@ scrape_configs:
- localhost
labels:
job: gerrit_sshd
__path__: #@ "{}/sshd_log.json".format(data.values.gerritServers[i].promtail.logPath)
host: #@ data.values.gerritServers[i].host
__path__: #@ "{}/sshd_log.json".format(data.values.gerritServers.other[i].promtail.logPath)
host: #@ data.values.gerritServers.other[i].host
entry_parser: raw
pipeline_stages:
- json:

View file

@ -231,10 +231,12 @@ def install(config_manager, output_dir, dryrun, update_repo):
namespace = config_manager.get_config()["namespace"]
_create_dashboard_configmaps(output_dir, namespace)
_create_promtail_configs(config, output_dir)
if os.path.exists(os.path.join(output_dir, "promtail.yaml")):
_create_promtail_configs(config, output_dir)
if not dryrun:
_download_promtail(output_dir)
if not dryrun:
_download_promtail(output_dir)
if update_repo:
_update_helm_repos()
_deploy_loose_resources(output_dir)