2024-10-07 19:41:23 +00:00
|
|
|
namespace: resources
|
2024-08-09 13:45:36 +00:00
|
|
|
groups:
|
2024-10-07 19:41:23 +00:00
|
|
|
- name: Host & hardware
|
|
|
|
|
rules:
|
|
|
|
|
- alert: HostOutOfMemory
|
|
|
|
|
expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10) * on (instance) group_left (nodename) node_uname_info{nodename=~".+"}
|
|
|
|
|
for: 2m
|
|
|
|
|
labels:
|
|
|
|
|
severity: warning
|
|
|
|
|
annotations:
|
|
|
|
|
description: |-
|
|
|
|
|
Node memory is filling up (< 10% left)
|
|
|
|
|
VALUE = {{ $value }}
|
|
|
|
|
LABELS = {{ $labels }}
|
|
|
|
|
summary: Host out of memory (instance {{ $labels.instance }})
|
|
|
|
|
- alert: HostMemoryUnderMemoryPressure
|
|
|
|
|
expr: (rate(node_vmstat_pgmajfault[1m]) > 1000) * on (instance) group_left (nodename) node_uname_info{nodename=~".+"}
|
|
|
|
|
for: 2m
|
|
|
|
|
labels:
|
|
|
|
|
severity: warning
|
|
|
|
|
annotations:
|
|
|
|
|
description: |-
|
|
|
|
|
The node is under heavy memory pressure. High rate of major page faults
|
|
|
|
|
VALUE = {{ $value }}
|
|
|
|
|
LABELS = {{ $labels }}
|
|
|
|
|
summary: Host memory under memory pressure (instance {{ $labels.instance }})
|
|
|
|
|
- alert: HostMemoryIsUnderutilized
|
|
|
|
|
expr: (100 - (avg_over_time(node_memory_MemAvailable_bytes[30m]) / node_memory_MemTotal_bytes * 100) < 20) * on (instance) group_left (nodename) node_uname_info{nodename=~".+"}
|
|
|
|
|
for: 1w
|
|
|
|
|
labels:
|
|
|
|
|
severity: info
|
|
|
|
|
annotations:
|
|
|
|
|
description: |-
|
|
|
|
|
Node memory is < 20% for 1 week. Consider reducing memory space. (instance {{ $labels.instance }})
|
|
|
|
|
VALUE = {{ $value }}
|
|
|
|
|
LABELS = {{ $labels }}
|
|
|
|
|
summary: Host Memory is underutilized (instance {{ $labels.instance }})
|
|
|
|
|
- alert: HostOutOfDiskSpace
|
|
|
|
|
expr: ((node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 10 and on (instance, device, mountpoint) node_filesystem_readonly == 0) * on (instance) group_left (nodename) node_uname_info{nodename=~".+"}
|
|
|
|
|
for: 2m
|
|
|
|
|
labels:
|
|
|
|
|
severity: warning
|
|
|
|
|
annotations:
|
|
|
|
|
description: |-
|
|
|
|
|
Disk is almost full (< 10% left)
|
|
|
|
|
VALUE = {{ $value }}
|
|
|
|
|
LABELS = {{ $labels }}
|
|
|
|
|
summary: Host out of disk space (instance {{ $labels.instance }})
|
|
|
|
|
- alert: HostDiskWillFillIn24Hours
|
|
|
|
|
expr: ((node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 10 and on (instance, device, mountpoint) predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs"}[1h], 24 * 3600) < 0 and on (instance, device, mountpoint) node_filesystem_readonly == 0) * on (instance) group_left (nodename) node_uname_info{nodename=~".+"}
|
|
|
|
|
for: 2m
|
|
|
|
|
labels:
|
|
|
|
|
severity: warning
|
|
|
|
|
annotations:
|
|
|
|
|
description: |-
|
|
|
|
|
Filesystem is predicted to run out of space within the next 24 hours at current write rate
|
|
|
|
|
VALUE = {{ $value }}
|
|
|
|
|
LABELS = {{ $labels }}
|
|
|
|
|
summary: Host disk will fill in 24 hours (instance {{ $labels.instance }})
|
|
|
|
|
- alert: HostCpuIsUnderutilized
|
|
|
|
|
expr: (100 - (rate(node_cpu_seconds_total{mode="idle"}[30m]) * 100) < 20) * on (instance) group_left (nodename) node_uname_info{nodename=~".+"}
|
|
|
|
|
for: 1w
|
|
|
|
|
labels:
|
|
|
|
|
severity: info
|
|
|
|
|
annotations:
|
|
|
|
|
description: |-
|
|
|
|
|
CPU load is < 20% for 1 week. Consider reducing the number of CPUs.
|
|
|
|
|
VALUE = {{ $value }}
|
|
|
|
|
LABELS = {{ $labels }}
|
|
|
|
|
summary: Host CPU is underutilized (instance {{ $labels.instance }})
|
|
|
|
|
- alert: HostCpuStealNoisyNeighbor
|
|
|
|
|
expr: (avg by (instance) (rate(node_cpu_seconds_total{mode="steal"}[5m])) * 100 > 10) * on (instance) group_left (nodename) node_uname_info{nodename=~".+"}
|
|
|
|
|
labels:
|
|
|
|
|
severity: warning
|
|
|
|
|
annotations:
|
|
|
|
|
description: |-
|
|
|
|
|
CPU steal is > 10%. A noisy neighbor is killing VM performances or a spot instance may be out of credit.
|
|
|
|
|
VALUE = {{ $value }}
|
|
|
|
|
LABELS = {{ $labels }}
|
|
|
|
|
summary: Host CPU steal noisy neighbor (instance {{ $labels.instance }})
|
|
|
|
|
- alert: HostOomKillDetected
|
|
|
|
|
expr: (increase(node_vmstat_oom_kill[1m]) > 0) * on (instance) group_left (nodename) node_uname_info{nodename=~".+"}
|
|
|
|
|
labels:
|
|
|
|
|
severity: warning
|
|
|
|
|
annotations:
|
|
|
|
|
description: |-
|
|
|
|
|
OOM kill detected
|
|
|
|
|
VALUE = {{ $value }}
|
|
|
|
|
LABELS = {{ $labels }}
|
|
|
|
|
summary: Host OOM kill detected (instance {{ $labels.instance }})
|
|
|
|
|
- alert: HostNetworkInterfaceSaturated
|
|
|
|
|
expr: ((rate(node_network_receive_bytes_total{device!~"^tap.*|^vnet.*|^veth.*|^tun.*"}[1m]) + rate(node_network_transmit_bytes_total{device!~"^tap.*|^vnet.*|^veth.*|^tun.*"}[1m])) / node_network_speed_bytes{device!~"^tap.*|^vnet.*|^veth.*|^tun.*"} > 0.8 < 10000) * on (instance) group_left (nodename) node_uname_info{nodename=~".+"}
|
|
|
|
|
for: 1m
|
|
|
|
|
labels:
|
|
|
|
|
severity: warning
|
|
|
|
|
annotations:
|
|
|
|
|
description: |-
|
|
|
|
|
The network interface "{{ $labels.device }}" on "{{ $labels.instance }}" is getting overloaded.
|
|
|
|
|
VALUE = {{ $value }}
|
|
|
|
|
LABELS = {{ $labels }}
|
|
|
|
|
summary: Host Network Interface Saturated (instance {{ $labels.instance }})
|
