190 lines
9.5 KiB
Markdown
190 lines
9.5 KiB
Markdown
|
# Monitoring setup for Gerrit
|
||
|
|
|
||
|
|
This project provides a setup for monitoring Gerrit instances. The setup is
|
||
|
|
based on Prometheus and Grafana running in Kubernetes. In addition, logging will
|
||
|
|
be provided by Grafana Loki.
|
||
|
|
|
||
|
|
The setup is provided as a helm chart. It can be installed using Helm
|
||
|
|
(This README expects Helm version 3.0 or higher).
|
||
|
|
|
||
|
|
The charts used in this setup are the chart provided in the open source and can be
|
||
|
|
found on GitHub:
|
||
|
|
|
||
|
|
- [Prometheus](https://github.com/helm/charts/tree/master/stable/prometheus)
|
||
|
|
- [Grafana](https://github.com/helm/charts/tree/master/stable/grafana)
|
||
|
|
- [Loki](https://github.com/grafana/loki/tree/master/production/helm/loki)
|
||
|
|
|
||
|
|
This project just provides `values.yaml`-files that are already configured to
|
||
|
|
work with the `metrics-reporter-prometheus`-plugin of Gerrit to make the setup
|
||
|
|
easier.
|
||
|
|
|
||
|
|
## Dependencies
|
||
|
|
|
||
|
|
- Gerrit \
|
||
|
|
Gerrit requires the following plugin to be installed:
|
||
|
|
- [metrics-reporter-prometheus](https://gerrit.googlesource.com/plugins/metrics-reporter-prometheus/)
|
||
|
|
|
||
|
|
- Promtail \
|
||
|
|
Promtail has to be installed with access to the `logs`-directory in the Gerrit-
|
||
|
|
site. A configuration-file for Promtail will be provided in this setup. Find
|
||
|
|
the documentation for Promtail
|
||
|
|
[here](https://github.com/grafana/loki/blob/master/docs/clients/promtail/README.md)
|
||
|
|
|
||
|
|
- Helm \
|
||
|
|
To install and configure Helm, follow the
|
||
|
|
[official guide](https://helm.sh/docs/intro/quickstart/#install-helm).
|
||
|
|
|
||
|
|
- ytt \
|
||
|
|
ytt is a templating tool for yaml-files. It is required for some last moment
|
||
|
|
configuration. Installation instructions can be found
|
||
|
|
[here](https://k14s.io/#install-from-github-release).
|
||
|
|
|
||
|
|
- yq \
|
||
|
|
yq is a commandline processor for yaml-files. Installation instructions can be
|
||
|
|
found [here](https://mikefarah.gitbook.io/yq/).
|
||
|
|
|
||
|
|
## Add dashboards
|
||
|
|
|
||
|
|
To have dashboards deployed automatically during installation, export the dashboards
|
||
|
|
to a JSON-file or create JSON-files describing the dashboards in another way.
|
||
|
|
Put these dashboards into the `./dashboards`-directory of this repository. During
|
||
|
|
the installation the dashboards will be added to a configmap and with this
|
||
|
|
automatically installed to Grafana.
|
||
|
|
|
||
|
|
## Configuration
|
||
|
|
|
||
|
|
While this project is supposed to provide a specialized and opinionated monitoring
|
||
|
|
setup, some configuration is highly dependent on the specific installation.
|
||
|
|
These options have to be configured in the `./config.yaml` before installing and
|
||
|
|
are listed here:
|
||
|
|
|
||
|
|
| option | description |
|
||
|
|
|-----------------------------------------|------------------------------------------------------------------------------------|
|
||
|
|
| `gerritServers.[0].host` | Hostname (incl. port, if required) of the Gerrit server to monitor |
|
||
|
|
| `gerritServers.[0].username` | Username of Gerrit user with 'View Metrics' capabilities |
|
||
|
|
| `gerritServers.[0].password` | Password of Gerrit user with 'View Metrics' capabilities |
|
||
|
|
| `namespace` | The namespace the charts are installed to |
|
||
|
|
| `tls.skipVerify` | Whether to skip TLS certificate verification |
|
||
|
|
| `tls.caCert` | CA certificate used for TLS certificate verification |
|
||
|
|
| `promtail.storagePath` | Path to directory, where Promtail is allowed to save files (e.g. `positions.yaml`) |
|
||
|
|
| `promtail.logPath` | Path to directory containing the Gerrit logs (e.g. `/var/gerrit/logs`) |
|
||
|
|
| `prometheus.server.host` | Prometheus server ingress hostname |
|
||
|
|
| `prometheus.server.username` | Username for Prometheus |
|
||
|
|
| `prometheus.server.password` | Password for Prometheus |
|
||
|
|
| `prometheus.server.tls.cert` | TLS certificate |
|
||
|
|
| `prometheus.server.tls.key` | TLS key |
|
||
|
|
| `prometheus.alertmanager.slack.apiUrl` | API URL of the Slack Webhook |
|
||
|
|
| `prometheus.alertmanager.slack.channel` | Channel to which the alerts should be posted |
|
||
|
|
| `loki.host` | Loki ingress hostname |
|
||
|
|
| `loki.username` | Username for Loki |
|
||
|
|
| `loki.password` | Password for Loki |
|
||
|
|
| `loki.tls.cert` | TLS certificate |
|
||
|
|
| `loki.tls.key` | TLS key |
|
||
|
|
| `grafana.host` | Grafana ingress hostname |
|
||
|
|
| `grafana.tls.cert` | TLS certificate |
|
||
|
|
| `grafana.tls.key` | TLS key |
|
||
|
|
| `grafana.admin.username` | Username for the admin user |
|
||
|
|
| `grafana.admin.password` | Password for the admin user |
|
||
|
|
| `grafana.ldap.enabled` | Whether to enable LDAP |
|
||
|
|
| `grafana.ldap.host` | Hostname of LDAP server |
|
||
|
|
| `grafana.ldap.port` | Port of LDAP server (Has to be `quoted`!) |
|
||
|
|
| `grafana.ldap.password` | Password of LDAP server |
|
||
|
|
| `grafana.ldap.bind_dn` | Bind DN (username) of the LDAP server |
|
||
|
|
| `grafana.ldap.accountBases` | List of base DNs to discover accounts (Has to have the format `"['a', 'b']"`) |
|
||
|
|
| `grafana.ldap.groupBases` | List of base DNs to discover groups (Has to have the format `"['a', 'b']"`) |
|
||
|
|
| `grafana.dashboards.editable` | Whether dashboards can be edited manually in the UI |
|
||
|
|
|
||
|
|
### Encryption
|
||
|
|
|
||
|
|
The configuration file contains secrets. Thus, to be able to share the configuration,
|
||
|
|
e.g. with the CI-system, it is meant to be encrypted. The encryption is explained
|
||
|
|
[here](./documentation/config-management.md).
|
||
|
|
|
||
|
|
The `./install.sh`-script will decrypt the file before templating, if it was
|
||
|
|
encrypted with `sops`.
|
||
|
|
|
||
|
|
## Installation
|
||
|
|
|
||
|
|
Before beginning with the installation, ensure that the local helm repository is
|
||
|
|
up-to-date:
|
||
|
|
|
||
|
|
```sh
|
||
|
|
helm repo add loki https://grafana.github.io/loki/charts
|
||
|
|
helm repo update
|
||
|
|
```
|
||
|
|
|
||
|
|
This project provides a script to quickly install the monitoring setup. To use
|
||
|
|
it, run:
|
||
|
|
|
||
|
|
```sh
|
||
|
|
./install.sh \
|
||
|
|
[--output ./dist] \
|
||
|
|
[--dryrun] \
|
||
|
|
config.yaml
|
||
|
|
```
|
||
|
|
|
||
|
|
The command will use the given configuration to create the final
|
||
|
|
files in the directory given by `--output` (default `./dist`) and install/update
|
||
|
|
the Kubernetes resources and charts, if the `--dryrun` flag is not set.
|
||
|
|
|
||
|
|
## Configure Promtail
|
||
|
|
|
||
|
|
Promtail has to be installed with access to the directory containing the Gerrit
|
||
|
|
logs, e.g. on the same host. The installation as described above will create a
|
||
|
|
configuration file for Promtail, which can be found in `./dist/promtail.yaml`.
|
||
|
|
Use it to configure Promtail by using the `-config.file=./dist/promtail.yaml`-
|
||
|
|
parameter, when starting Promtail. Using the Promtail binary directly this would
|
||
|
|
result in the following command:
|
||
|
|
|
||
|
|
```sh
|
||
|
|
$PATH_TO_PROMTAIL/promtail \
|
||
|
|
-config.file=./dist/promtail.yaml \
|
||
|
|
-client.external-labels=host=$(hostname)
|
||
|
|
```
|
||
|
|
|
||
|
|
The `-client.external-labels=host=$(hostname)` option will add a label to each job
|
||
|
|
that contains the hostname. This is useful, if multiple host are scraped for logs
|
||
|
|
and only one Grafana is used to view the logs.
|
||
|
|
|
||
|
|
If TLS-verification is activated, the CA-certificate used for verification
|
||
|
|
(usually the one configured for `tls.caCert`) has to be present in the
|
||
|
|
directory configured for `promtail.storagePath` in the `config.yaml` and has to
|
||
|
|
be called `promtail.ca.crt`.
|
||
|
|
|
||
|
|
The Promtail configuration provided here expects the logs to be available in
|
||
|
|
JSON-format. This can be configured by setting `log.jsonLogging = true` in the
|
||
|
|
`gerrit.config`.
|
||
|
|
|
||
|
|
## Uninstallation
|
||
|
|
|
||
|
|
To remove the Prometheus chart from the cluster, run
|
||
|
|
|
||
|
|
```sh
|
||
|
|
helm uninstall prometheus --namespace $NAMESPACE
|
||
|
|
helm uninstall loki --namespace $NAMESPACE
|
||
|
|
helm uninstall grafana --namespace $NAMESPACE
|
||
|
|
kubectl delete -f ./dist/configuration
|
||
|
|
```
|
||
|
|
|
||
|
|
To also release the volumes, run
|
||
|
|
|
||
|
|
```sh
|
||
|
|
kubectl delete -f ./dist/storage
|
||
|
|
```
|
||
|
|
|
||
|
|
NOTE: Doing so, all data, which was not backed up will be lost!
|
||
|
|
|
||
|
|
Remove the namespace:
|
||
|
|
|
||
|
|
```sh
|
||
|
|
kubectl delete -f ./dist/namespace.yaml
|
||
|
|
```
|
||
|
|
|
||
|
|
The `./uninstall.sh`-script will automatically remove the charts installed in
|
||
|
|
by the `./install.sh`-script from the configured namespace and delete the
|
||
|
|
namespace as well:
|
||
|
|
|
||
|
|
```sh
|
||
|
|
./uninstall.sh config.yaml
|
||
|
|
```
|