Skip to content

XMS Observability Stack

1. Introduction

This document describes the XMS Observability Stack, a solution specifically designed to provide deep insights into the operational status and performance of the XoT Management System (XMS) and the devices connected through it.

The XMS Observability Stack leverages the following components:

  • Prometheus: Collects time-series metrics from XMS components and potentially configured endpoints on connected devices. This provides crucial data on resource utilization, request rates, error counts, and other performance indicators specific to the XMS environment.
  • Loki: Aggregates logs generated by XMS services and forwarded via FluentBit. By indexing metadata (labels) rather than full log content, Loki enables efficient querying and analysis of events, errors, and operational messages from across the XMS system.
  • Grafana: Serves as the central visualization and dashboard platform. It connects to Prometheus (for metrics) and Loki (for logs), presenting a unified view of XMS health. Dashboards allow operators to correlate metrics and logs, quickly identify anomalies, and diagnose issues within XMS or related to connected device interactions.

By deploying this stack, XMS administrators gain the ability to:

  • Monitor the real-time health and performance of the XMS infrastructure.
  • Analyze trends and identify potential bottlenecks before they impact service.
  • Efficiently troubleshoot issues by correlating metrics and logs from various XMS components.
  • Gain visibility into the status and activity of devices managed by XMS, based on the data XMS collects or forwards.

2. Setup

2.1 Requirements

  • Docker and Docker Compose
  • DNS entries for web, loki, prom and grafana.EXAMPLE.DOMAIN.COM

2.2 Configuration

  • Clone XMS Observability repo or expand a downloaded archive in an appropriate location e.g under '/opt/' or '/srv/'
  • Create new certificates or copy in existing ones for nginx under the folder credentials.
  • Create client in XMS Keycloak (e.g. https://xms.example.domain.com/idp), under Realm XMS -> Clients -> Create Client
  • Set: Client ID: grafana-oauth (or similar), Name: Grafana Client (free text) , Description: can be empty
  • Client authentication: On, Authorization: Off, Standard FLow & Direct Access Grants should be enabled, rest is off
  • Root URL: https://web.grafana.EXAMPLE.DOMAIN.COM/ ,
  • Home URL: empty
  • Valid redirect URIs: https://web.grafana.EXAMPLE.DOMAIN.COM/login/generic_oauth,
  • Valid logout: empty
  • Web origins: https://web.grafana.EXAMPLE.DOMAIN.COM/,
  • Admin URL: https://web.grafana.EXAMPLE.DOMAIN.COM/
  • Generate a client secret in the Credentials tab
  • In client scopes, assign 'none'
  • Edit configuration files:
  • grafana.ini
    • URL as Root URL above
    • Admin Password, set a random string there.
    • Grafana client secret that was generated in Keycloak
    • Update auth_url, token_url and api_url.
  • nginx.conf.d/service.conf
  • Create password for xms user in nginx: sudo htpasswd ./httpasswd xms in '/usr/share/xms-observability'
  • apache2-utils might need to be installed for the command to be present
  • added later to loki.conf and prometheus.conf in xms server logging configuration

2.3 XMS Configuration

  • Add grafana variables in XMS docker-compose.yaml in dashboard container declaration:
  • GRAFANA_ENABLED=true
  • GRAFANA_URL=https://web.grafana.EXAMPLE.DOMAIN.COM/d/
  • GRAFANA_ENVIRONMENT=ENVIRONMENT_VARIABLE (output of hostname)
  • GRAFANA_XOT_DASHBOARD_PATH={$DASHBOARD_ID}/individual-xot-dashboard
  • Extra topic in benthos.yaml
 input:
  label: ""
  mqtt:
    urls: ["mqtt:1883"]
    topics: ["xot/+/v2/log_entry","xotclient/+/v2/log_entry"]
    client_id: "xot-log-collector"

2.3.1 Install FluentBit

FluentBit is used to forward events and logs from XMS to the observability stack.

  • curl https://packages.fluentbit.io/fluentbit.key | gpg --dearmor > /usr/share/keyrings/fluentbit-keyring.gpg
  • add to /etc/apt/sources.list.d/fluentbit.list
  • deb [signed-by=/usr/share/keyrings/fluentbit-keyring.gpg] https://packages.fluentbit.io/ubuntu/${CODENAME} ${CODENAME} main
  • remove original /etc/fluent-bit/fluent-bit.conf
  • create new with just following line:
  • @INCLUDE /etc/xms/config/logging/fluent-bit.conf
  • start end enable fluentbit with systemctl
    • systemctl start fluentbit --enable
  • if we are using self-signed certs, tls.verify should be set to off inside
    • /etc/xms/config/logging/prometheus.conf
    • /etc/xms/config/logging/loki.conf

2.3.2 Logging Configuration

  • Add extra in /usr/share/xms/config/logging/fluent-bit.conf
[SERVICE]
    HTTP_Server On
    HTTP_Listen 127.0.0.1
    HTTP_Port 2020

@INCLUDE /etc/xms/config/logging/loki.conf
@INCLUDE /etc/xms/config/logging/prometheus.conf
  • /usr/share/xms/config/logging/loki.conf (aka output_loki.conf)
[OUTPUT]
    name        loki
    match       logs.*
    host        loki.EXAMPLE.DOMAIN.COM
    uri         /loki/api/v1/push
    port        443
    tls         on
    tls.verify  on
    http_user   xms
    http_passwd PASSWORD
    line_format json
    labels      job=$HOSTNAME, source=$SOURCE, level=$level
  • Add to /usr/share/xms/config/logging/fluent-bit.conf
[INPUT]
    name            prometheus_scrape
    host            127.0.0.1
    port            9090
    metrics_path    /mqttmetrics
    tag             metrics.collect
    scrape_interval 10s

and populate Host, http_user and http_password in OUTPUT section with prom.EXAMPLE.DOMAIN.COM, xms and xms password respectively.

3. Initialize

Start docker compose in the directory where the docker-compose.yaml is placed: sudo docker compose up -d . Most likely the XMS needs to be also restarted in order for the extra variables to come into effect. This can be achieved with sudo systemctl restart xms on the console of XMS server.

Afterwards, a button will appear in the diagnostics page of every device in the XMS web interface that will point to the Grafana URL that was configured earlier.