XMS Observability Stack

Overview

The XMS Observability Stack provides monitoring, logging, and visualization for the XoT Management System (XMS) and its connected devices. It combines Prometheus (metrics), Loki (log aggregation), and Grafana (dashboards) to give operators a unified view of system health.

With this stack, XMS administrators can:

• Monitor real-time health and performance of XMS infrastructure.
• Analyze trends and identify potential bottlenecks before they impact service.
• Troubleshoot issues by correlating metrics and logs from various XMS components.
• Gain visibility into the status and activity of devices managed by XMS.

Prerequisites

Requirement	Details
Operating System	Ubuntu (Noble or Jammy)
Docker & Docker Compose	Required for Docker Compose deployment
DNS entries	web.grafana.<DOMAIN>, grafana.<DOMAIN>, loki.<DOMAIN>, prom.<DOMAIN>
SSL certificates	Let's Encrypt (recommended) or self-signed (auto-generated for development)

Deployment

Choose one of the two deployment options below.

Option A: Docker Compose

1. Clone or extract the repository

Clone the XMS Observability repo, or expand the compressed copy from here into an appropriate location (e.g., /opt/ or /srv/).

1. Configure environment variables

cp .env.example .env

Edit .env with your domain and credentials. By default, subdomains are web.grafana, grafana, loki, and prom. To customize, set prefix overrides in .env (e.g., LOKI_PREFIX=logs produces logs.<DOMAIN>).

1. Generate configuration files

./scripts/init.sh

This reads .env and generates grafana.ini and nginx.conf.d/service.conf from their .template files. Re-run this script whenever you change .env.

init.sh also auto-generates htpasswd from BASIC_AUTH_USER and BASIC_AUTH_PASSWORD in .env. To regenerate manually, delete the existing file and re-run init.sh, or use htpasswd directly (apache2-utils required). These credentials are used in the FluentBit Loki and Prometheus configs on XMS servers.

For development, init.sh auto-generates self-signed certificates if none exist at credentials/fullchain.pem and credentials/privkey.pem.

1. Start the stack

sudo docker compose up -d

Option B: Debian Package

The Debian package installs the stack to /opt/xms-observability/ with systemd integration.

1. Install the package

Download from https://xertpubrepo.blob.core.windows.net/v38/xms-observability-*.deb and install like this:

sudo apt install ./xms-observability_*.deb

The installer collects configuration values via interactive debconf prompts, generates configs from templates via envsubst, creates htpasswd, generates self-signed certs if needed, and enables a systemd service.

1. Service management

systemctl status xms-observability
systemctl restart xms-observability
systemctl stop xms-observability

Post-Deployment Configuration

SSL with Let's Encrypt

For production deployments, Let's Encrypt is strongly recommended over self-signed certificates.

1. Obtain certificates — Run the automated setup script:

sudo ./scripts/setup-certbot.sh

This obtains certificates via certbot with webroot challenge mode and copies them into credentials/. The port 80 Nginx server block serves ACME challenge tokens from /var/www/certbot/ before redirecting other traffic to HTTPS.

Note: If you already have certificates, you can skip this step and place them manually at credentials/fullchain.pem and credentials/privkey.pem.

1. Verify certificates are in place — Confirm both files exist:

ls -l credentials/fullchain.pem credentials/privkey.pem

1. Automatic renewal — The setup script installs a deploy hook for automatic renewal via certbot.timer. Verify the timer is active:

systemctl status certbot.timer

Keycloak OAuth2 (Optional)

OAuth is disabled by default. To enable it, first create a client in XMS Keycloak (e.g., https://xms.<DOMAIN>/idp), under Realm XMS > Clients > Create Client.

Client Settings

Field	Value
Client ID	grafana-oauth (or similar)
Name	Grafana Client (free text)
Description	(can be empty)
Client authentication	On
Authorization	Off
Standard Flow	Enabled
Direct Access Grants	Enabled
Root URL	https://web.grafana.<DOMAIN>/
Home URL	(empty)
Valid redirect URIs	https://web.grafana.<DOMAIN>/login/generic_oauth
Valid logout URIs	(empty)
Web origins	https://web.grafana.<DOMAIN>/
Admin URL	https://web.grafana.<DOMAIN>/

After creating the client:

1. Generate a client secret in the Credentials tab.

2. In Client Scopes, assign none.

3. Run the setup script:

./scripts/setup-oauth.sh

The script prompts for Keycloak URL, client ID, and client secret, then updates .env and regenerates grafana.ini. It can be re-run at any time to reconfigure or disable OAuth (--disable flag).

XMS Integration

This section covers configuration on the XMS servers that send data to the observability stack.

Environment Variables

Add the following to the XMS server's .env file:

GRAFANA_ENABLED=true
GRAFANA_URL=https://web.grafana.<DOMAIN>/d/
GRAFANA_ENVIRONMENT=<hostname>
GRAFANA_XOT_DASHBOARD_PATH=b0167a72-3cc2-4eee-99aa-0a53f06e61a1/individual-xot-dashboard

Variable	Description
GRAFANA_ENABLED	Enables the Grafana integration in the XMS web interface.
GRAFANA_URL	Base URL for Grafana dashboard links.
GRAFANA_ENVIRONMENT	A label identifying this XMS instance (typically the output of hostname). Used to filter dashboards by environment.
GRAFANA_XOT_DASHBOARD_PATH	Path to the individual XoT device dashboard in Grafana.

Restart XMS to apply the changes:

sudo systemctl restart xms

FluentBit Setup

1. Add the GPG key

curl https://packages.fluentbit.io/fluentbit.key | gpg --dearmor | sudo tee /usr/share/keyrings/fluentbit-keyring.gpg > /dev/null

1. Add the APT repository

Add the following to /etc/apt/sources.list.d/fluentbit.list, replacing ${CODENAME} with noble or jammy:

deb [signed-by=/usr/share/keyrings/fluentbit-keyring.gpg] https://packages.fluentbit.io/ubuntu/${CODENAME} ${CODENAME} main

1. Install FluentBit

sudo apt update && sudo apt install fluent-bit

1. Configure FluentBit for XMS

Replace the default config to include XMS logging configuration:

sudo cp /etc/fluent-bit/fluent-bit.conf /etc/fluent-bit/fluent-bit.conf.bak
echo '@INCLUDE /etc/xms/config/logging/fluent-bit.conf' | sudo tee /etc/fluent-bit/fluent-bit.conf

1. Start the service

sudo systemctl enable --now fluent-bit

1. TLS verification settings

If using self-signed certificates, set tls.verify to off in:

• /etc/xms/config/logging/prometheus.conf
• /etc/xms/config/logging/loki.conf

Warning: Disabling TLS verification removes certificate validation. This is acceptable for development but should be avoided in production. Use Let's Encrypt certificates for production deployments.

If using Let's Encrypt certificates, no changes are needed — tls.verify can remain on.

Verification

After completing deployment and XMS integration, verify that the stack is working correctly:

1. Check that all services are running

sudo docker compose ps        # Docker Compose deployment
systemctl status xms-observability  # Debian package deployment

1. Access Grafana — Navigate to https://web.grafana.<DOMAIN>/ and log in with the credentials configured in .env.

2. Confirm Prometheus targets — In Grafana, go to the Prometheus data source and verify that XMS targets are listed and in an UP state.

3. Verify logs in Loki — Use the Explore view in Grafana with the Loki data source to confirm logs are being received from the XMS servers.

4. Check the XMS web interface — Open the diagnostics page for any device in XMS. A Grafana button should appear linking to the device's dashboard.

Troubleshooting

Symptom	Possible Cause	Resolution
Grafana is not accessible	Nginx misconfiguration or DNS not resolving	Verify DNS entries resolve correctly. Check Nginx logs: sudo docker compose logs nginx
Prometheus targets show DOWN	FluentBit not running or wrong basic auth credentials	Verify FluentBit is running (systemctl status fluent-bit). Confirm BASIC_AUTH_USER/BASIC_AUTH_PASSWORD in .env match the FluentBit config.
No logs appearing in Loki	FluentBit misconfigured or TLS verification failing	Check FluentBit logs: journalctl -u fluent-bit. If using self-signed certs, ensure tls.verify is set to off in the Loki output config.
Grafana button missing in XMS	Environment variables not applied	Confirm GRAFANA_ENABLED=true is set in the XMS .env file and restart XMS: sudo systemctl restart xms.
OAuth login fails	Keycloak client misconfiguration	Verify redirect URIs and client secret match. Re-run ./scripts/setup-oauth.sh to reconfigure.
Certificate renewal not working	Certbot timer not active	Check timer status: systemctl status certbot.timer. Verify the webroot path /var/www/certbot/ is accessible on port 80.