XMS Observability Stack

Introduction

This document describes the XMS Observability Stack, a solution specifically designed to provide deep insights into the operational status and performance of the XoT Management System (XMS) and the devices connected through it.

The XMS Observability Stack leverages the following components:

Prometheus: Collects time-series metrics from XMS components and potentially configured endpoints on connected devices. This provides crucial data on resource utilization, request rates, error counts, and other performance indicators specific to the XMS environment.
Loki: Aggregates logs generated by XMS services and forwarded via FluentBit. By indexing metadata (labels) rather than full log content, Loki enables efficient querying and analysis of events, errors, and operational messages from across the XMS system.
Grafana: Serves as the central visualization and dashboard platform. It connects to Prometheus (for metrics) and Loki (for logs), presenting a unified view of XMS health. Dashboards allow operators to correlate metrics and logs, quickly identify anomalies, and diagnose issues within XMS or related to connected device interactions.

By deploying this stack, XMS administrators gain the ability to:

Monitor the real-time health and performance of the XMS infrastructure.
Analyze trends and identify potential bottlenecks before they impact service.
Efficiently troubleshoot issues by correlating metrics and logs from various XMS components.
Gain visibility into the status and activity of devices managed by XMS, based on the data XMS collects or forwards.

Setup

Requirements

Docker and Docker Compose
DNS entries for web, loki, prom and grafana.EXAMPLE.DOMAIN.COM

Configuration

Clone XMS Observability repo or expand the compressed copy from here in an appropriate location e.g under '/opt/' or '/srv/'
Create new certificates or copy in existing ones for nginx under the folder credentials. Consider using certbot for this.
Create client in XMS Keycloak (e.g. https://xms.example.domain.com/idp), under Realm XMS -> Clients -> Create Client
Set: Client ID: grafana-oauth (or similar), Name: Grafana Client (free text) , Description: can be empty
Client authentication: On, Authorization: Off, Standard FLow & Direct Access Grants should be enabled, rest is off
Root URL: https://web.grafana.EXAMPLE.DOMAIN.COM/ ,
Home URL: empty
Valid redirect URIs: https://web.grafana.EXAMPLE.DOMAIN.COM/login/generic_oauth,
Valid logout: empty
Web origins: https://web.grafana.EXAMPLE.DOMAIN.COM/,
Admin URL: https://web.grafana.EXAMPLE.DOMAIN.COM/
Generate a client secret in the Credentials tab
In client scopes, assign 'none'
Edit configuration files:
grafana.ini
- URL as Root URL above
- Admin Password, set a random string there.
- Grafana client secret that was generated in Keycloak
- Update auth_url, token_url and api_url.
nginx.conf.d/service.conf
Create password for xms user in nginx: sudo htpasswd ./httpasswd xms in '/usr/share/xms-observability'
apache2-utils might need to be installed for the command to be present
added later to loki.conf and prometheus.conf in xms server logging configuration

XMS Configuration

Add grafana variables in XMS docker-compose.yaml in dashboard container declaration:

GRAFANA_ENABLED=true

GRAFANA_URL=https://web.grafana.EXAMPLE.DOMAIN.COM/d/

GRAFANA_ENVIRONMENT=ENVIRONMENT_VARIABLE (output of hostname)

GRAFANA_XOT_DASHBOARD_PATH={$DASHBOARD_ID}/individual-xot-dashboard

Extra topic in benthos.yaml

 input:
  label: ""
  mqtt:
    urls: ["mqtt:1883"]
    topics: ["xot/+/v2/log_entry","xotclient/+/v2/log_entry"]
    client_id: "xot-log-collector"

Install FluentBit

FluentBit is used to forward events and logs from XMS to the observability stack.

curl https://packages.fluentbit.io/fluentbit.key | gpg --dearmor > /usr/share/keyrings/fluentbit-keyring.gpg
add to /etc/apt/sources.list.d/fluentbit.list the following, where CODENAME can be noble or jammy.
deb [signed-by=/usr/share/keyrings/fluentbit-keyring.gpg] https://packages.fluentbit.io/ubuntu/${CODENAME} ${CODENAME} main
Install fluent-bit: sudo apt install fluent-bit
remove original /etc/fluent-bit/fluent-bit.conf

cp /etc/fluent-bit/fluent-bit.conf /etc/fluent-bit/fluent-bit.conf.bak
echo "@INCLUDE /etc/xms/config/logging/fluent-bit.conf" > /etc/fluent-bit/fluent-bit.conf

start end enable fluentbit with systemctl
systemctl enable --now fluent-bit
if we are using self-signed certs, tls.verify should be set to off inside
/etc/xms/config/logging/prometheus.conf
/etc/xms/config/logging/loki.conf

Logging Configuration

Add extra in /usr/share/xms/config/logging/fluent-bit.conf

[SERVICE]
    HTTP_Server On
    HTTP_Listen 127.0.0.1
    HTTP_Port 2020

@INCLUDE /etc/xms/config/logging/loki.conf
@INCLUDE /etc/xms/config/logging/prometheus.conf

/usr/share/xms/config/logging/loki.conf (aka output_loki.conf)

[OUTPUT]
    name        loki
    match       logs.*
    host        loki.EXAMPLE.DOMAIN.COM
    uri         /loki/api/v1/push
    port        443
    tls         on
    tls.verify  on
    http_user   xms
    http_passwd PASSWORD
    line_format json
    labels      job=$HOSTNAME, source=$SOURCE, level=$level

Add to /usr/share/xms/config/logging/fluent-bit.conf

[INPUT]
    name            prometheus_scrape
    host            127.0.0.1
    port            9090
    metrics_path    /mqttmetrics
    tag             metrics.collect
    scrape_interval 10s

and populate Host, http_user and http_password in OUTPUT section with prom.EXAMPLE.DOMAIN.COM, xms and xms password respectively.

Initialize

Start docker compose in the directory where the docker-compose.yaml is placed: sudo docker compose up -d . Most likely the XMS needs to be also restarted in order for the extra variables to come into effect. This can be achieved with sudo systemctl restart xms on the console of XMS server.

Afterwards, a button will appear in the diagnostics page of every device in the XMS web interface that will point to the Grafana URL that was configured earlier.