Home Server Setup Guide (Part 6): Monitoring Your Server with Prometheus and Grafana on NixOS

Welcome to my tech blog on Glitch! In this post, I’ll walk you through setting up Prometheus and Grafana on a NixOS server to monitor system health metrics like CPU, memory, disk, and network usage. This setup is perfect for my home lab, where I run services like Pi-hole and Vaultwarden, and it’s accessible remotely via Tailscale. I’ll break it down step-by-step with examples, explain why each component matters, and show how I got it running. Let’s dive in!

Prometheus is an open-source, time-series-based monitoring tool designed for collecting and querying metrics. It uses a pull-based model, scraping metrics from HTTP endpoints (like Node Exporter) and storing them in a time-series database. Its query language, PromQL, lets you analyze metrics in real-time. Prometheus is perfect for dynamic environments like Kubernetes or, in my case, a home lab server.

• Scrapes metrics from services and exporters (e.g., Node Exporter for system stats).
  
• Supports alerting via Alertmanager (e.g., for Slack or email notifications).
  
• Lightweight and integrates seamlessly with Grafana.

Example: In my setup, Prometheus scrapes system metrics (CPU, memory) from Node Exporter, giving me a full picture of my server’s health.

Grafana is an open-source visualization platform that turns raw metrics into beautiful, interactive dashboards. It pulls data from sources like Prometheus and lets you create graphs, charts, and alerts.

• Supports multiple data sources (Prometheus, MySQL, etc.).
• Customizable dashboards with a drag-and-drop UI.
• Community templates (like the Node Exporter Full dashboard) for quick setup.

Example: I use Grafana to visualize my server’s CPU usage accessible remotely via Tailscale

Monitoring ensures your server and services (like Pi-hole or Vaultwarden) are running smoothly. With Node Exporter, Prometheus, and Grafana, I can:

• Track CPU, memory, disk, and network usage.
• Detect performance bottlenecks or crashes.
• Visualize trends over time (e.g., spikes in DNS queries).
• Access dashboards securely via Tailscale.

Node Exporter is a Prometheus component that collects system-level metrics (e.g., CPU, memory, disk I/O) and exposes them via http://localhost:9100/metrics

Without Node Exporter, Prometheus can only scrape application-specific metrics (e.g., Pi-hole). Node Exporter gives you visibility into your server’s overall health.

Add the following to /etc/nixos/configuration.nix:

services.prometheus.exporters.node = {
  enable = true;
  port = 9100;
};

networking.firewall.allowedTCPPorts = [ 9100 ];

Rebuild your NixOS system:

sudo nixos-rebuild switch

Check the service status:

systemctl status prometheus-node-exporter

curl http://localhost:9100/metrics

You should see a long list of metrics like node_cpu_seconds_total, node_memory_MemAvailable_bytes, etc.

Prometheus pulls metrics from Node Exporter and stores them for querying. We’ll configure it to scrape localhost:9100 and run its web UI on port 9090.

Add this to /etc/nixos/configuration.nix:

services.prometheus = {
  enable = true;
  port = 9090;
  scrapeConfigs = [
    {
      job_name = "node";
      static_configs = [
        { targets = [ "localhost:9100" ]; }
      ];
    }
  ];
};

networking.firewall.allowedTCPPorts = [ 9090 ];

Rebuild the system:

sudo nixos-rebuild switch

Check the service status:

systemctl status prometheus

Access the Prometheus UI at http://Tailscale-IP:9090 (via Tailscale). Go to Status > Targets and confirm the node job (localhost:9100) is in the UP state.

Grafana visualizes Prometheus metrics in interactive dashboards. We’ll run it on port 3000 and make it accessible via Tailscale.

Add this to /etc/nixos/configuration.nix:

services.grafana = {
  enable = true;
  settings.server = {
    http_addr = "0.0.0.0"; # Accessible from any interface
    http_port = 3000;
  };
};

networking.firewall.allowedTCPPorts = [ 3000 ];

Rebuild the system:

sudo nixos-rebuild switch

Check the service status:

systemctl status grafana

Access Grafana at http://Tailscale-IP:3000 (via Tailscale). Log in

• In Grafana, go to Connections > Data sources.
• Click Add data source and select Prometheus.
• Set the URL to http://localhost:9090.
• Click Save & test to confirm the connection.

• CPU usage by core
• Memory and swap usage
• Disk I/O and space
• Network throughput
• System uptime

• Add more exporters to monitor application-specific metrics.
• Set up Prometheus Alertmanager for notifications.
• Explore Grafana’s advanced features like custom dashboards or annotations.

Setting up Prometheus and Grafana on NixOS transformed how I monitor my home lab. With Node Exporter, I get detailed system metrics, Prometheus stores and queries them, and Grafana makes them easy to visualize. Whether you’re running a simple server or a complex Kubernetes cluster, this stack is a game-changer. Try it out,let me know how it goes on the linkedin!

Happy monitoring! 🚀