This also removes the fixed versions in the Pipfile. Since versions are
already locked in the Pipfile.lock, this was not necessary and only
made things harder to manage.
Change-Id: I3ce60de4178f9647b82af3c32800bca5f369a456
Using a local PGP-key for encryption of the secrets in the configuration
is not very secure and makes it hard to rotate and distribute the
key. Sops provides the option to use managed services for this
purpose, e.g. HashiCorp Vault.
This change adds the option to use HashiCorp Vault, when using the
provided python scripts to encrypt the config file.
Change-Id: I7683fbfdbed00506c3bca264ac8565f48bc5ea73
Gerrit instances that are loadbalanced cannot easily e scraped by
an external Prometheus, since the request won't end up at a specified
Gerrit instance. A typical setup to solve this issue, is to install a
local Prometheus and scrape the local Prometheus from the central
Prometheus. This is a so called federated setup.
Now such a setup is supported and can be configured.
Change-Id: I0119d3c1d846cd8e975e5732f4d59cf863c6d2b8
Some dashboards were still explicitly specifying 'Prometheus' as the
datasource, which leads to issues when trying to import the dashboards
into a grafana instance where the prometheus datasource has a different
name.
Change-Id: I13135af32a6f312a4feb32ab828f906f7b13edfe
The healthcheck plugin for Gerrit provides a convenient way to determine
the health of different functionalities and components of Gerrit. If
the endpoint provided by the plugin is pinged, it will execute a set
of checks and return either 200 if all checks passed or 500 if at least
one failed. It will also provide metrics that can be scraped by
Prometheus.
This change adds the option for Gerrit installations outside of Kubernetes
to install a sidecar container in the Prometheus deployment that every
30 s pings the healthcheck plugin's endpoint, thereby triggering the
checks. This is not provided for kubernetes, since there the ping should
be the task of the Kubernetes liveness probes.
The change additionally adds a dashboard displaying the status of the
healthcheck for each Gerrit instance over time.
Change-Id: Ieeedc4406b642e542c89679a8314d771ca0928af
Grafonnet doesn't yet provide addSeriesOverrides() accepting an array.
Also use a different color for each gc so that switching to another
gc shows up in the graph.
Change-Id: I4e424280d44a63f57ad7196dfdb7e77ba2f13f24
This updates the Grafana chart to the new repository, since the old
repository is now deprecated. This also updates the container images
and Grafana version.
Change-Id: I29e38d7c23bfa95992537efae7b8b3967d71ffd0
There are a lot of latency metrics. This change splits up the existing
dashboard for latencies. For REST API latencies, it also allows to
select the REST API calls to look at. This change also adds latency
dashboards for the NoteDB and UI Actions.
Change-Id: Idb9631cc1bc838d06e626d58f163e71fb78b30c5
Versioning the pure JSON files representing the Grafana dashboards
had some disadvantages. It was hard to review them, they were very
cluttered and a lot was duplicated.
There are some tools that deal with that. One of them is Grafonnet,
which is a superset of Jsonnet, a tool to create JSON files using a
domain specific language.
This change implements the Gerrit Process dashboard in Grafonnet.
It also extends the installer to be able to install dashboards in
the Jsonnet format.
Change-Id: I6235fb7d045bd71557678a4e3b0d4ad4515f4615
This also changes the helm chart repository, since the old one was
deprecated. Further, the new version adapts the resources to not
contain deprecated APIs.
Change-Id: Idd3f1ed48e22da303fd62d9c2ee63ccb959ed948
Grafana provides a repository for dashboards that can be used to easily
import dashboards. Providing these dashboards there would make it easier
for users not using the full setup provided here to still use the
dashboards. To be able to upload however, the datasource reference in the
dashboards has to be a template.
This is however not compatible with the way how the dashboards are imported
in the Grafana of the stack provided here. Thus, the variables are
removed during the installation.
Change-Id: I99f127882a6f7594ca1c40fbe1e299378e89f4e9
This change
- adds metrics for parallel GC to the GC panel in the Gerrit Process
dashboard
- configures the GC panel to only show queries with values other than
null
- changes the interval to one minute, which fits the scrape interval
- changes the default time frame to the last 24h, which is used for
most other dashboards
Change-Id: I3b6587e769ae7486a02e26b8d7f2822319eb94e6
The promtail chart is anyway configured to use the Loki service for
pushing logs. The service itself is not password protected and this
was thus not required.
Change-Id: I886b76ca7e5d6e8af370a2cd0f527892008c7600
* changes:
Adapt to ytt 0.28.0
Sort monitoring and logging components into sub-maps in the config
Collect logs from Gerrit in Kubernetes
Add promtail chart to collect logs from cluster
Ytt 0.28.0 introduced a breaking change. The --output-directory
option was removed. This was done because this option implicitly
emptied the directory, which could be dangerous. While this option
still exist under a different name, the --output-files option is
now recommended.
The installer now uses the --output-files option, but to ensure a
clean installation, it checks, whether the directory already exists
and if it does, asks the user, whether it can empty it. If it is
not allowed to do so, the installation will abort.
Change-Id: I574c3b054e9293c0534d609c062946cd39890793
This adds a service discovery configuration for promtail to also
collect logs for Gerrit installations in Kubernetes. The installations
will be discovered by namespace and a given label.
Change-Id: I894e47f37428add9b44df6596950d314ee2a3ed0
This adds the promtail chart to the installation that allows to
collect the logs of the applications in the cluster, which are written
to stdout of the containers.
This will only collect logs from pods in the same namespace as the
monitoring setup. In a later change also logs from Gerrit instances
in Kubernetes will be added.
Change-Id: I86c5c5470eaa31191fb5ac339ee21dee85106097
So far it was only possible to monitor single instance Gerrit servers.
This was due to to the fact that a URL had to be used that pointed to
a dedicated instance, since if multiple replicas would be behind the
instance, the metrics of a random replica would be scraped and not of
all.
Prometheus has a service discovery functionality for deployments running
in Kubernetes. This is now used, when monitoring a Gerrit instance in
Kubernetes. This allows to have a variable number of replicas running,
which will be automatically discovered by Prometheus.
The dashboards were adapted accordingly and allow now to select the
replica to be observed. For now, no summary of all replicas can be
displayed in the dashboards, but that feature is planned to be added
in the future.
Change-Id: I96efc63a192cd90f5e3e91a53dace8e1ae83132e
This replaces the hacky graph showing the Gerrit version with a table
showing the current Gerrit version information.
Change-Id: Idfbdc85e376953aead40fea06544e5c84fb777e7
Add graphs for the following latency metrics
- receive-commit
- query total
- query changes
- REST total
- REST change list comments
- REST change list robot comments
- REST change post review
- REST get change detail
- REST get change diff
- REST get change
- REST get commit
- REST get change revision actions
Change-Id: Id782e12335ae76820cac4e4e8c80450671bf8216
The installation failed, if TLS verification was disabled and no CA
certificate was given in the configuration. This happened because the
installation script always expected the CA certificate.
The installation now only expects the certificate, if TLS verification
is enabled.
Change-Id: I5429fc1ee0d230c74cc0689607cf2736d6520030
This adds the promtail version used in the setup to a file and adds
an installation step downloading promtail, if the installation is not
run in `dryrun`-mode.
Change-Id: I1127220a57b2610b5c4458ce2205077706a860e6
So far the install-script could only create a single promtail config.
Since the monitoring setup is able to monitor multiple Gerrit servers,
this caused manual work to create a promtail config per Gerrit server.
Now ytt will create a configuration for each Gerrit host configured
in the config.yaml. Ytt is only able to do that in a single file. Thus,
csplit is used to split the files into separate files that can then
be used to configure promtail on the respective hosts. The config-
files can then be found under
$OUTPUT/promtail/promtail-$GERRIT_HOSTNAME.yaml.
Change-Id: Ib09fba83d8a8fbd45b42e9e5388a85a37ab1a952
The scripts were written in bash. Using bash became quite unwieldy.
Python by nature can deal well with yaml and is thus better suited
in dealing with the yaml-based configuration files. This change
rewrites the original scripts staying as close as possible to the
original ones.
Right now, the python scripts call subprocesses a lot to work with
the tools, which were already used before. At least for yaml-
templating there may be better tools that have a python integration,
which could be used in the future.
Change-Id: Ida16318445a05dcfdada9c7a56a391e4827f02e7
raito
