the-distro/gerrit-monitoring
0
0
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity

Merge changes I574c3b05,I95020080,I894e47f3,I86c5c547

Browse source 
* changes:
  Adapt to ytt 0.28.0
  Sort monitoring and logging components into sub-maps in the config
  Collect logs from Gerrit in Kubernetes
  Add promtail chart to collect logs from cluster
This commit is contained in:
Thomas Dräbing 2020-06-30 12:51:50 +00:00 committed by Gerrit Code Review
parent ad0b8c71ee 89ee46a081
commit 50c3a5aac8
20 changed files with 448 additions and 122 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

77
README.md
View file

@ -10,9 +10,10 @@ The setup is provided as a helm chart. It can be installed using Helm
The charts used in this setup are the chart provided in the open source and can be The charts used in this setup are the chart provided in the open source and can be
found on GitHub: found on GitHub:
- [Prometheus](https://github.com/helm/charts/tree/master/stable/prometheus)
- [Grafana](https://github.com/helm/charts/tree/master/stable/grafana) - [Grafana](https://github.com/helm/charts/tree/master/stable/grafana)
- [Loki](https://github.com/grafana/loki/tree/master/production/helm/loki) - [Loki](https://github.com/grafana/loki/tree/master/production/helm/loki)
- [Prometheus](https://github.com/helm/charts/tree/master/stable/prometheus)
- [Promtail](https://github.com/grafana/loki/tree/master/production/helm/promtail)
This project just provides `values.yaml`-files that are already configured to This project just provides `values.yaml`-files that are already configured to
work with the `metrics-reporter-prometheus`-plugin of Gerrit to make the setup work with the `metrics-reporter-prometheus`-plugin of Gerrit to make the setup
@ -76,43 +77,43 @@ setup, some configuration is highly dependent on the specific installation.
These options have to be configured in the `./config.yaml` before installing and These options have to be configured in the `./config.yaml` before installing and
are listed here: are listed here:
| option | description | | option | description |
|-----------------------------------------|----------------------------------------------------------------------------------------| |----------------------------------------------------|----------------------------------------------------------------------------------------|
| `gerritServers` | List of Gerrit servers to scrape. For details refer to section [below](#gerritServers) | | `gerritServers` | List of Gerrit servers to scrape. For details refer to section [below](#gerritServers) |
| `namespace` | The namespace the charts are installed to | | `namespace` | The namespace the charts are installed to |
| `tls.skipVerify` | Whether to skip TLS certificate verification | | `tls.skipVerify` | Whether to skip TLS certificate verification |
| `tls.caCert` | CA certificate used for TLS certificate verification | | `tls.caCert` | CA certificate used for TLS certificate verification |
| `prometheus.server.host` | Prometheus server ingress hostname | | `monitoring.prometheus.server.host` | Prometheus server ingress hostname |
| `prometheus.server.username` | Username for Prometheus | | `monitoring.prometheus.server.username` | Username for Prometheus |
| `prometheus.server.password` | Password for Prometheus | | `monitoring.prometheus.server.password` | Password for Prometheus |
| `prometheus.server.tls.cert` | TLS certificate | | `monitoring.prometheus.server.tls.cert` | TLS certificate |
| `prometheus.server.tls.key` | TLS key | | `monitoring.prometheus.server.tls.key` | TLS key |
| `prometheus.alertmanager.slack.apiUrl` | API URL of the Slack Webhook | | `monitoring.prometheus.alertmanager.slack.apiUrl` | API URL of the Slack Webhook |
| `prometheus.alertmanager.slack.channel` | Channel to which the alerts should be posted | | `monitoring.prometheus.alertmanager.slack.channel` | Channel to which the alerts should be posted |
| `loki.host` | Loki ingress hostname | | `monitoring.grafana.host` | Grafana ingress hostname |
| `loki.username` | Username for Loki | | `monitoring.grafana.tls.cert` | TLS certificate |
| `loki.password` | Password for Loki | | `monitoring.grafana.tls.key` | TLS key |
| `loki.s3.protocol` | Protocol used for communicating with S3 | | `monitoring.grafana.admin.username` | Username for the admin user |
| `loki.s3.host` | Hostname of the S3 object store | | `monitoring.grafana.admin.password` | Password for the admin user |
| `loki.s3.accessToken` | The EC2 accessToken used for authentication with S3 | | `monitoring.grafana.ldap.enabled` | Whether to enable LDAP |
| `loki.s3.secret` | The secret associated with the accessToken | | `monitoring.grafana.ldap.host` | Hostname of LDAP server |
| `loki.s3.bucket` | The name of the S3 bucket | | `monitoring.grafana.ldap.port` | Port of LDAP server (Has to be `quoted`!) |
| `loki.s3.region` | The region in which the S3 bucket is hosted | | `monitoring.grafana.ldap.password` | Password of LDAP server |
| `loki.tls.cert` | TLS certificate | | `monitoring.grafana.ldap.bind_dn` | Bind DN (username) of the LDAP server |
| `loki.tls.key` | TLS key | | `monitoring.grafana.ldap.accountBases` | List of base DNs to discover accounts (Has to have the format `"['a', 'b']"`) |
| `grafana.host` | Grafana ingress hostname | | `monitoring.grafana.ldap.groupBases` | List of base DNs to discover groups (Has to have the format `"['a', 'b']"`) |
| `grafana.tls.cert` | TLS certificate | | `monitoring.grafana.dashboards.editable` | Whether dashboards can be edited manually in the UI |
| `grafana.tls.key` | TLS key | | `logging.loki.host` | Loki ingress hostname |
| `grafana.admin.username` | Username for the admin user | | `logging.loki.username` | Username for Loki |
| `grafana.admin.password` | Password for the admin user | | `logging.loki.password` | Password for Loki |
| `grafana.ldap.enabled` | Whether to enable LDAP | | `logging.loki.s3.protocol` | Protocol used for communicating with S3 |
| `grafana.ldap.host` | Hostname of LDAP server | | `logging.loki.s3.host` | Hostname of the S3 object store |
| `grafana.ldap.port` | Port of LDAP server (Has to be `quoted`!) | | `logging.loki.s3.accessToken` | The EC2 accessToken used for authentication with S3 |
| `grafana.ldap.password` | Password of LDAP server | | `logging.loki.s3.secret` | The secret associated with the accessToken |
| `grafana.ldap.bind_dn` | Bind DN (username) of the LDAP server | | `logging.loki.s3.bucket` | The name of the S3 bucket |
| `grafana.ldap.accountBases` | List of base DNs to discover accounts (Has to have the format `"['a', 'b']"`) | | `logging.loki.s3.region` | The region in which the S3 bucket is hosted |
| `grafana.ldap.groupBases` | List of base DNs to discover groups (Has to have the format `"['a', 'b']"`) | | `logging.loki.tls.cert` | TLS certificate |
| `grafana.dashboards.editable` | Whether dashboards can be edited manually in the UI | | `logging.loki.tls.key` | TLS key |
### `gerritServers` ### `gerritServers`

4
cfgmgr/abstract.py
View file

@ -26,8 +26,8 @@ class AbstractConfigManager(abc.ABC):
self.config_path = config_path self.config_path = config_path
self.requires_htpasswd = [ self.requires_htpasswd = [
["loki"], ["logging", "loki"],
["prometheus", "server"], ["monitoring", "prometheus", "server"],
] ]
def get_config(self): def get_config(self):

2
charts/grafana/configuration/grafana.ca.secret.yaml
View file

@ -1,6 +1,6 @@
#@ load("@ytt:data", "data") #@ load("@ytt:data", "data")
#@ load("@ytt:base64", "base64") #@ load("@ytt:base64", "base64")
#@ if data.values.grafana.ldap.enabled and not data.values.tls.skipVerify: #@ if data.values.monitoring.grafana.ldap.enabled and not data.values.tls.skipVerify:
apiVersion: v1 apiVersion: v1
kind: Secret kind: Secret
metadata: metadata:

6
charts/grafana/configuration/grafana.secret.yaml
View file

@ -7,9 +7,9 @@ metadata:
name: grafana-credentials name: grafana-credentials
namespace: #@ data.values.namespace namespace: #@ data.values.namespace
data: data:
admin-user: #@ base64.encode(data.values.grafana.admin.username) admin-user: #@ base64.encode(data.values.monitoring.grafana.admin.username)
admin-password: #@ base64.encode(data.values.grafana.admin.password) admin-password: #@ base64.encode(data.values.monitoring.grafana.admin.password)
#@ if data.values.grafana.ldap.enabled: #@ if data.values.monitoring.grafana.ldap.enabled:
ldap-toml: #@ base64.encode(format_ldap_toml()) ldap-toml: #@ base64.encode(format_ldap_toml())
#@ end #@ end
type: Opaque type: Opaque

4
charts/grafana/configuration/grafana.tls.secret.yaml
View file

@ -7,5 +7,5 @@ metadata:
namespace: #@ data.values.namespace namespace: #@ data.values.namespace
type: kubernetes.io/tls type: kubernetes.io/tls
data: data:
tls.crt: #@ base64.encode(data.values.grafana.tls.cert) tls.crt: #@ base64.encode(data.values.monitoring.grafana.tls.cert)
tls.key: #@ base64.encode(data.values.grafana.tls.key) tls.key: #@ base64.encode(data.values.monitoring.grafana.tls.key)

12
charts/grafana/configuration/ldap.lib.txt
View file

@ -2,18 +2,18 @@
(@ def format_ldap_toml(): -@) (@ def format_ldap_toml(): -@)
[[servers]] [[servers]]
host = "(@= data.values.grafana.ldap.host @)" host = "(@= data.values.monitoring.grafana.ldap.host @)"
port = (@= data.values.grafana.ldap.port @) port = (@= data.values.monitoring.grafana.ldap.port @)
use_ssl = true use_ssl = true
start_tls = false start_tls = false
ssl_skip_verify = (@= "{}".format(data.values.tls.skipVerify).lower() @) ssl_skip_verify = (@= "{}".format(data.values.tls.skipVerify).lower() @)
root_ca_cert = "/etc/secrets/server.ca.crt" root_ca_cert = "/etc/secrets/server.ca.crt"
bind_dn = "(@= data.values.grafana.ldap.bind_dn @)" bind_dn = "(@= data.values.monitoring.grafana.ldap.bind_dn @)"
bind_password = "(@= data.values.grafana.ldap.password @)" bind_password = "(@= data.values.monitoring.grafana.ldap.password @)"
search_filter = "(cn=%s)" search_filter = "(cn=%s)"
search_base_dns = (@= data.values.grafana.ldap.accountBases @) search_base_dns = (@= data.values.monitoring.grafana.ldap.accountBases @)
group_search_filter = "(cn=%s)" group_search_filter = "(cn=%s)"
group_search_base_dns = (@= data.values.grafana.ldap.groupBases @) group_search_base_dns = (@= data.values.monitoring.grafana.ldap.groupBases @)
[[servers.group_mappings]] [[servers.group_mappings]]
group_dn = "*" group_dn = "*"

15
charts/grafana/grafana.yaml
View file

@ -101,7 +101,8 @@ downloadDashboards:
# podAnnotations: {} # podAnnotations: {}
## Pod Labels ## Pod Labels
# podLabels: {} podLabels:
app: grafana
podPortName: grafana podPortName: grafana
@ -129,7 +130,7 @@ ingress:
labels: {} labels: {}
path: / path: /
hosts: hosts:
- #@ data.values.grafana.host - #@ data.values.monitoring.grafana.host
## Extra paths to prepend to every host configuration. This is useful when working with annotation based services. ## Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
extraPaths: [] extraPaths: []
# - path: /* # - path: /*
@ -139,7 +140,7 @@ ingress:
tls: tls:
- secretName: grafana-server-tls - secretName: grafana-server-tls
hosts: hosts:
- #@ data.values.grafana.host - #@ data.values.monitoring.grafana.host
resources: resources:
limits: limits:
@ -270,7 +271,7 @@ envRenderSecret: {}
## Additional grafana server secret mounts ## Additional grafana server secret mounts
# Defines additional mounts with secrets. Secrets must be manually created in the namespace. # Defines additional mounts with secrets. Secrets must be manually created in the namespace.
extraSecretMounts: extraSecretMounts:
#@ if data.values.grafana.ldap.enabled and not data.values.tls.skipVerify: #@ if data.values.monitoring.grafana.ldap.enabled and not data.values.tls.skipVerify:
- name: tls-ca - name: tls-ca
mountPath: /etc/secrets mountPath: /etc/secrets
secretName: grafana-ca secretName: grafana-ca
@ -395,7 +396,7 @@ grafana.ini:
## LDAP Authentication can be enabled with the following values on grafana.ini ## LDAP Authentication can be enabled with the following values on grafana.ini
## NOTE: Grafana will fail to start if the value for ldap.toml is invalid ## NOTE: Grafana will fail to start if the value for ldap.toml is invalid
auth.ldap: auth.ldap:
enabled: #@ data.values.grafana.ldap.enabled enabled: #@ data.values.monitoring.grafana.ldap.enabled
allow_sign_up: true allow_sign_up: true
config_file: /etc/grafana/ldap.toml config_file: /etc/grafana/ldap.toml
@ -405,7 +406,7 @@ grafana.ini:
## ref: http://docs.grafana.org/installation/configuration/#auth-ldap ## ref: http://docs.grafana.org/installation/configuration/#auth-ldap
## ref: http://docs.grafana.org/installation/ldap/#configuration ## ref: http://docs.grafana.org/installation/ldap/#configuration
ldap: ldap:
enabled: #@ data.values.grafana.ldap.enabled enabled: #@ data.values.monitoring.grafana.ldap.enabled
# `existingSecret` is a reference to an existing secret containing the ldap configuration # `existingSecret` is a reference to an existing secret containing the ldap configuration
# for Grafana in a key `ldap-toml`. # for Grafana in a key `ldap-toml`.
existingSecret: "grafana-credentials" existingSecret: "grafana-credentials"
@ -474,7 +475,7 @@ sidecar:
# disableDelete to activate a import-only behaviour # disableDelete to activate a import-only behaviour
disableDelete: true disableDelete: true
# allow updating provisioned dashboards from the UI # allow updating provisioned dashboards from the UI
allowUiUpdates: #@ data.values.grafana.dashboards.editable allowUiUpdates: #@ data.values.monitoring.grafana.dashboards.editable
datasources: datasources:
enabled: false enabled: false
## Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH requests, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds. ## Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH requests, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds.

2
charts/loki/configuration/loki.basic-auth.secret.yaml
View file

@ -6,5 +6,5 @@ metadata:
name: loki-basic-auth name: loki-basic-auth
namespace: #@ data.values.namespace namespace: #@ data.values.namespace
data: data:
auth: #@ base64.encode(data.values.loki.htpasswd) auth: #@ base64.encode(data.values.logging.loki.htpasswd)
type: Opaque type: Opaque

4
charts/loki/configuration/loki.tls.secret.yaml
View file

@ -7,5 +7,5 @@ metadata:
namespace: #@ data.values.namespace namespace: #@ data.values.namespace
type: kubernetes.io/tls type: kubernetes.io/tls
data: data:
tls.crt: #@ base64.encode(data.values.loki.tls.cert) tls.crt: #@ base64.encode(data.values.logging.loki.tls.cert)
tls.key: #@ base64.encode(data.values.loki.tls.key) tls.key: #@ base64.encode(data.values.logging.loki.tls.key)

8
charts/loki/loki.yaml
View file

@ -14,13 +14,13 @@ ingress:
nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required' nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required'
# kubernetes.io/tls-acme: "true" # kubernetes.io/tls-acme: "true"
hosts: hosts:
- host: #@ data.values.loki.host - host: #@ data.values.logging.loki.host
paths: paths:
- / - /
tls: tls:
- secretName: loki-server-tls - secretName: loki-server-tls
hosts: hosts:
- #@ data.values.loki.host - #@ data.values.logging.loki.host
## Affinity for pod assignment ## Affinity for pod assignment
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
@ -87,7 +87,7 @@ config:
boltdb: boltdb:
directory: /data/loki/index directory: /data/loki/index
aws: aws:
s3: #@ "{}://{}:{}@{}/{}".format(data.values.loki.s3.protocol, data.values.loki.s3.accessToken, data.values.loki.s3.secret, data.values.loki.s3.host, data.values.loki.s3.bucket) s3: #@ "{}://{}:{}@{}/{}".format(data.values.logging.loki.s3.protocol, data.values.logging.loki.s3.accessToken, data.values.logging.loki.s3.secret, data.values.logging.loki.s3.host, data.values.logging.loki.s3.bucket)
s3forcepathstyle: true s3forcepathstyle: true
chunk_store_config: chunk_store_config:
max_look_back_period: 0 max_look_back_period: 0
@ -243,4 +243,4 @@ extraPorts: []
# Extra env variables to pass to the loki container # Extra env variables to pass to the loki container
env: env:
- name: AWS_REGION - name: AWS_REGION
value: #@ data.values.loki.s3.region value: #@ data.values.logging.loki.s3.region

2
charts/prometheus/configuration/prometheus.basic-auth.secret.yaml
View file

@ -6,5 +6,5 @@ metadata:
name: prometheus-basic-auth name: prometheus-basic-auth
namespace: #@ data.values.namespace namespace: #@ data.values.namespace
data: data:
auth: #@ base64.encode(data.values.prometheus.server.htpasswd) auth: #@ base64.encode(data.values.monitoring.prometheus.server.htpasswd)
type: Opaque type: Opaque

4
charts/prometheus/configuration/prometheus.secret.yaml
View file

@ -17,7 +17,7 @@ data:
#@ if not data.values.tls.skipVerify: #@ if not data.values.tls.skipVerify:
server.ca.crt: #@ base64.encode(data.values.tls.caCert) server.ca.crt: #@ base64.encode(data.values.tls.caCert)
server.crt: #@ base64.encode(data.values.prometheus.server.tls.cert) server.crt: #@ base64.encode(data.values.monitoring.prometheus.server.tls.cert)
server.key: #@ base64.encode(data.values.prometheus.server.tls.key) server.key: #@ base64.encode(data.values.monitoring.prometheus.server.tls.key)
#@ end #@ end
type: Opaque type: Opaque

4
charts/prometheus/configuration/prometheus.tls.secret.yaml
View file

@ -7,5 +7,5 @@ metadata:
namespace: #@ data.values.namespace namespace: #@ data.values.namespace
type: kubernetes.io/tls type: kubernetes.io/tls
data: data:
tls.crt: #@ base64.encode(data.values.prometheus.server.tls.cert) tls.crt: #@ base64.encode(data.values.monitoring.prometheus.server.tls.cert)
tls.key: #@ base64.encode(data.values.prometheus.server.tls.key) tls.key: #@ base64.encode(data.values.monitoring.prometheus.server.tls.key)

8
charts/prometheus/prometheus.yaml
View file

@ -675,7 +675,7 @@ server:
## Must be provided if Ingress is enabled ## Must be provided if Ingress is enabled
## ##
hosts: hosts:
- #@ data.values.prometheus.server.host - #@ data.values.monitoring.prometheus.server.host
# - prometheus.domain.com # - prometheus.domain.com
# - domain.com/prometheus # - domain.com/prometheus
@ -692,7 +692,7 @@ server:
tls: tls:
- secretName: prometheus-server-tls - secretName: prometheus-server-tls
hosts: hosts:
- #@ data.values.prometheus.server.host - #@ data.values.monitoring.prometheus.server.host
## Server Deployment Strategy type ## Server Deployment Strategy type
# strategy: # strategy:
@ -1055,12 +1055,12 @@ pushgateway:
alertmanagerFiles: alertmanagerFiles:
alertmanager.yml: alertmanager.yml:
global: global:
slack_api_url: #@ data.values.prometheus.alertmanager.slack.apiUrl slack_api_url: #@ data.values.monitoring.prometheus.alertmanager.slack.apiUrl
receivers: receivers:
- name: gerrit-admin - name: gerrit-admin
slack_configs: slack_configs:
- channel: #@ data.values.prometheus.alertmanager.slack.channel - channel: #@ data.values.monitoring.prometheus.alertmanager.slack.channel
send_resolved: true send_resolved: true
title: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}" title: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"
text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}" text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"

1
charts/promtail/VERSION Normal file
View file

@ -0,0 +1 @@
0.22.1

300
charts/promtail/promtail.yaml Normal file
View file

@ -0,0 +1,300 @@
#@ load("@ytt:data", "data")
## Affinity for pod assignment
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
affinity: {}
annotations: {}
# The update strategy to apply to the DaemonSet
##
deploymentStrategy: {}
# rollingUpdate:
# maxUnavailable: 1
# type: RollingUpdate
initContainer:
enabled: false
fsInotifyMaxUserInstances: 128
image:
repository: grafana/promtail
tag: v1.3.0
pullPolicy: IfNotPresent
livenessProbe: {}
loki:
serviceName: #@ "loki-{}".format(data.values.namespace)
servicePort: 3100
serviceScheme: http
user: #@ data.values.logging.loki.username
password: #@ data.values.logging.loki.password
nameOverride: #@ "promtail-{}".format(data.values.namespace, data.values.namespace)
## Node labels for pod assignment
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
nodeSelector: {}
pipelineStages:
- docker: {}
## Pod Labels
podLabels: {}
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "http-metrics"
## Assign a PriorityClassName to pods if set
# priorityClassName:
rbac:
create: true
pspEnabled: true
readinessProbe:
failureThreshold: 5
httpGet:
path: /ready
port: http-metrics
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 200m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
# Custom scrape_configs to override the default ones in the configmap
scrapeConfigs:
- job_name: kubernetes-pods-monitoring
pipeline_stages:
- docker: {}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: labeldrop
regex: '__meta_kubernetes_pod_label_app_kubernetes_io_.*'
- action: labeldrop
regex: '__meta_kubernetes_pod_label_statefulset_kubernetes_io_.*'
- action: labeldrop
regex: '__meta_kubernetes_pod_label_controller_revision_hash'
- action: labeldrop
regex: '__meta_kubernetes_pod_label_pod_template_.*'
- source_labels:
- __meta_kubernetes_pod_label_name
target_label: __service__
- source_labels:
- __meta_kubernetes_pod_node_name
target_label: __host__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
replacement: $1
separator: /
source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_label_app
- __service__
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: keep
regex: #@ data.values.namespace
source_labels:
- namespace
- action: replace
source_labels:
- release
target_label: chart_release
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: instance
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container_name
- replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
#@ for gerrit in data.values.gerritServers.kubernetes:
- job_name: #@ "kubernetes-pods-gerrit-{}".format(gerrit.namespace)
pipeline_stages:
- docker: {}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: labeldrop
regex: '__meta_kubernetes_pod_label_pod_template_.*'
- source_labels:
- __meta_kubernetes_pod_label_name
target_label: __service__
- source_labels:
- __meta_kubernetes_pod_node_name
target_label: __host__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
replacement: $1
separator: /
source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_label_app
- __service__
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: keep
regex: #@ gerrit.namespace
source_labels:
- namespace
- source_labels:
- #@ "__meta_kubernetes_pod_label_{}".format(gerrit.label.name)
regex: #@ gerrit.label.value
action: keep
- action: replace
source_labels:
- release
target_label: chart_release
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: instance
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container_name
- replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
#@ end
# Custom scrape_configs together with the default ones in the configmap
extraScrapeConfigs: []
securityContext:
readOnlyRootFilesystem: true
runAsGroup: 0
runAsUser: 0
serviceAccount:
create: true
name: promtail
## Tolerations for pod assignment
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
# Extra volumes to scrape logs from
volumes:
- name: docker
hostPath:
path: /var/lib/docker/containers
- name: pods
hostPath:
path: /var/log/pods
# Custom volumes together with the default ones
extraVolumes: []
volumeMounts:
- name: docker
mountPath: /var/lib/docker/containers
readOnly: true
- name: pods
mountPath: /var/log/pods
readOnly: true
# Custom volumeMounts together with the default ones
extraVolumeMounts: []
# Add extra Commandline args while starting up promtail.
# more info : https://github.com/grafana/loki/pull/1530
extraCommandlineArgs: []
# example:
# extraCommandlineArgs:
# - -client.external-labels=hostname=$(HOSTNAME)
config:
client:
# Maximum wait period before sending batch
batchwait: 1s
# Maximum batch size to accrue before sending, unit is byte
batchsize: 102400
# Maximum time to wait for server to respond to a request
timeout: 10s
backoff_config:
# Initial backoff time between retries
minbackoff: 100ms
# Maximum backoff time between retries
maxbackoff: 5s
# Maximum number of retries when sending batches, 0 means infinite retries
maxretries: 20
# The labels to add to any time series or alerts when communicating with loki
external_labels: {}
server:
http_listen_port: 3101
positions:
filename: /run/promtail/positions.yaml
target_config:
# Period to resync directories being watched and files being tailed
sync_period: 10s
serviceMonitor:
enabled: false
interval: ""
additionalLabels: {}
# scrapeTimeout: 10s
# Extra env variables to pass to the promtail container
env: []
# enable and configure if using the syslog scrape config
syslogService:
enabled: false
type: ClusterIP
port: 1514
## Specify the nodePort value for the LoadBalancer and NodePort service types.
## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport
##
# nodePort:
## Provide any additional annotations which may be required. This can be used to
## set the LoadBalancer service type to internal only.
## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer
##
annotations: {}
labels: {}
## Use loadBalancerIP to request a specific static IP,
## otherwise leave blank
##
loadBalancerIP:
# loadBalancerSourceRanges: []
## Set the externalTrafficPolicy in the Service to either Cluster or Local
# externalTrafficPolicy: Cluster

82
config.yaml
View file

@ -19,47 +19,49 @@ namespace: namespace
tls: tls:
skipVerify: true skipVerify: true
caCert: caCert:
prometheus: monitoring:
server: prometheus:
host: prometheus.example.com server:
username: host: prometheus.example.com
password: username:
password:
tls:
cert:
key:
alertmanager:
slack:
apiUrl: https://hooks.slack.com/services/xxx/xxx
channel: '#alerts'
grafana:
host: grafana.example.com
tls: tls:
cert: cert:
key: key:
alertmanager: admin:
slack: username: admin
apiUrl: https://hooks.slack.com/services/xxx/xxx password: secret
channel: '#alerts' ldap:
loki: enabled: false
host: loki.example.com host:
username: port: ""
password: password:
s3: bind_dn:
protocol: https accountBases: "[]"
host: s3.eu-de-1.example.com groupBases: "[]"
accessToken: abcd dashboards:
secret: "1234" editable: false
bucket: bucket logging:
region: eu-de-1 loki:
tls: host: loki.example.com
cert: username:
key:
grafana:
host: grafana.example.com
tls:
cert:
key:
admin:
username: admin
password: secret
ldap:
enabled: false
host:
port: ""
password: password:
bind_dn: s3:
accountBases: "[]" protocol: https
groupBases: "[]" host: s3.eu-de-1.example.com
dashboards: accessToken: abcd
editable: false secret: "1234"
bucket: bucket
region: eu-de-1
tls:
cert:
key:

6
promtail/promtail.yaml → promtail/promtailLocalConfig.yaml
View file

@ -10,15 +10,15 @@ positions:
filename: #@ "{}/positions.yaml".format(data.values.gerritServers.other[i].promtail.storagePath) filename: #@ "{}/positions.yaml".format(data.values.gerritServers.other[i].promtail.storagePath)
clients: clients:
- url: #@ "https://{}/loki/api/v1/push".format(data.values.loki.host) - url: #@ "https://{}/loki/api/v1/push".format(data.values.logging.loki.host)
tls_config: tls_config:
insecure_skip_verify: #@ data.values.tls.skipVerify insecure_skip_verify: #@ data.values.tls.skipVerify
#@ if not data.values.tls.skipVerify: #@ if not data.values.tls.skipVerify:
ca_file: #@ "{}/promtail.ca.crt".format(data.values.gerritServers.other[i].promtail.storagePath) ca_file: #@ "{}/promtail.ca.crt".format(data.values.gerritServers.other[i].promtail.storagePath)
#@ end #@ end
basic_auth: basic_auth:
username: #@ data.values.loki.username username: #@ data.values.logging.loki.username
password: #@ data.values.loki.password password: #@ data.values.logging.loki.password
scrape_configs: scrape_configs:
- job_name: gerrit_error - job_name: gerrit_error
static_configs: static_configs:

1
subcommands/_globals.py
View file

@ -16,4 +16,5 @@ HELM_CHARTS = {
"grafana": "stable/grafana", "grafana": "stable/grafana",
"loki": "loki/loki", "loki": "loki/loki",
"prometheus": "stable/prometheus", "prometheus": "stable/prometheus",
"promtail": "loki/promtail",
} }

28
subcommands/install.py
View file

@ -14,7 +14,9 @@
import os.path import os.path
import stat import stat
import shutil
import subprocess import subprocess
import sys
import zipfile import zipfile
import requests import requests
@ -26,6 +28,7 @@ from ._globals import HELM_CHARTS
TEMPLATES = [ TEMPLATES = [
"charts/namespace.yaml", "charts/namespace.yaml",
"charts/prometheus", "charts/prometheus",
"charts/promtail",
"charts/loki", "charts/loki",
"charts/grafana", "charts/grafana",
"promtail", "promtail",
@ -79,7 +82,7 @@ def _create_promtail_configs(config, output_dir):
if not os.path.exists(os.path.join(output_dir, "promtail")): if not os.path.exists(os.path.join(output_dir, "promtail")):
os.mkdir(os.path.join(output_dir, "promtail")) os.mkdir(os.path.join(output_dir, "promtail"))
with open(os.path.join(output_dir, "promtail.yaml")) as f: with open(os.path.join(output_dir, "promtailLocalConfig.yaml")) as f:
for promtail_config in yaml.load_all(f, Loader=yaml.SafeLoader): for promtail_config in yaml.load_all(f, Loader=yaml.SafeLoader):
with open( with open(
os.path.join( os.path.join(
@ -94,7 +97,7 @@ def _create_promtail_configs(config, output_dir):
) as f: ) as f:
yaml.dump(promtail_config, f) yaml.dump(promtail_config, f)
os.remove(os.path.join(output_dir, "promtail.yaml")) os.remove(os.path.join(output_dir, "promtailLocalConfig.yaml"))
if not config["tls"]["skipVerify"]: if not config["tls"]["skipVerify"]:
try: try:
@ -145,7 +148,7 @@ def _run_ytt(config, output_dir):
command += ["-f", template] command += ["-f", template]
command += [ command += [
"--output-directory", "--output-files",
output_dir, output_dir,
"--ignore-unknown-comments", "--ignore-unknown-comments",
"-f", "-f",
@ -229,13 +232,30 @@ def install(config_manager, output_dir, dryrun, update_repo):
if not os.path.exists(output_dir): if not os.path.exists(output_dir):
os.mkdir(output_dir) os.mkdir(output_dir)
elif os.listdir(output_dir):
while True:
response = input(
(
"Output directory already exists. This may lead to file conflicts "
"and unwanted configuration applied to the cluster. Do you want "
"to empty the directory? [y/n] "
)
)
if response == "y":
shutil.rmtree(output_dir)
os.mkdir(output_dir)
break
if response == "n":
print("Aborting installation. Please provide empty directory.")
sys.exit(1)
print("Unknown input.")
_run_ytt(config, output_dir) _run_ytt(config, output_dir)
namespace = config_manager.get_config()["namespace"] namespace = config_manager.get_config()["namespace"]
_create_dashboard_configmaps(output_dir, namespace) _create_dashboard_configmaps(output_dir, namespace)
if os.path.exists(os.path.join(output_dir, "promtail.yaml")): if os.path.exists(os.path.join(output_dir, "promtailLocalConfig.yaml")):
_create_promtail_configs(config, output_dir) _create_promtail_configs(config, output_dir)
if not dryrun: if not dryrun:
_download_promtail(output_dir) _download_promtail(output_dir)