Commit graph

8 commits

Author SHA1 Message Date
Thomas Dräbing 50c3a5aac8 Merge changes I574c3b05,I95020080,I894e47f3,I86c5c547
* changes:
  Adapt to ytt 0.28.0
  Sort monitoring and logging components into sub-maps in the config
  Collect logs from Gerrit in Kubernetes
  Add promtail chart to collect logs from cluster
2020-06-30 12:51:50 +00:00
Thomas Draebing ad0b8c71ee Add alert on Gerrit threads in deadlock
This adds an alert that is firing, if 1 or more threads of a Gerrit
instance are in a deadlock.

Change-Id: Ie2e14e81381e07de2559b42b91d6e483639831ef
2020-06-25 09:00:06 +02:00
Thomas Draebing 3b4005a047 Sort monitoring and logging components into sub-maps in the config
This is done in preparation to allow multiple logging stacks.

Change-Id: I950200805ec01851bfdf6ccc3a5243893a947616
2020-05-27 16:30:33 +02:00
Thomas Draebing 451882b7e9 Allow to monitor Gerrit on Kubernetes
So far it was only possible to monitor single instance Gerrit servers.
This was due to to the fact that a URL had to be used that pointed to
a dedicated instance, since if multiple replicas would be behind the
instance, the metrics of a random replica would be scraped and not of
all.

Prometheus has a service discovery functionality for deployments running
in Kubernetes. This is now used, when monitoring a Gerrit instance in
Kubernetes. This allows to have a variable number of replicas running,
which will be automatically discovered by Prometheus.

The dashboards were adapted accordingly and allow now to select the
replica to be observed. For now, no summary of all replicas can be
displayed in the dashboards, but that feature is planned to be added
in the future.

Change-Id: I96efc63a192cd90f5e3e91a53dace8e1ae83132e
2020-05-14 15:55:35 +02:00
Thomas Draebing a8135ce8c4 Relabel the instance label for prometheus and loki metrics
The instance label for Prometheus had the value localhost:9090, which
was misleading.

Now the label is relabeled to prometheus-<namespace> or loki-<namespace>.
This is still not ideal for cases, where multiple replicas are deployed,
but until then, it is already a slight improvement.

Change-Id: I1efdc49071b1d3bf99d21315ca03821e9d58c906
2020-04-03 13:36:34 +02:00
Thomas Draebing b1be26012b Scrape Loki metrics
Change-Id: I2cd9c872882cd760fc2ff10028b7e03a31f8fba5
2020-03-23 16:09:54 +01:00
Thomas Draebing ead4e7d5cc Monitor Prometheus itself
Monitoring Prometheus itself will help to identify issues with the
monitoring setup itself.

Change-Id: I26cfd395831aebffe9f32922c8e795f8df928b9e
2020-03-23 15:39:29 +01:00
Thomas Draebing be862d863e Move internal project to open source
This change adds the current status of a project that aims to create
a simple monitoring setup to monitor Gerrit servers, which was developed
internally at SAP.

The project provides an opinionated and basic configuration for helm
charts that can be used to install Loki, Prometheus and Grafana on a
Kubernetes cluster. Scripts to easily apply the configuration and
install the whole setup are provided as well.

The contributions so far were done by (with number of commits)

  80  Thomas Draebing
  11  Matthias Sohn
   2  Saša Živkov

Change-Id: I8045780446edfb3c0dc8287b8f494505e338e066
2020-03-11 15:23:19 +01:00