Icinga Prometheus Remote Writer

                                

        
        

            
            

                

    

    
Icinga Prometheus Remote Writer


A simple bridge to bring Icinga 2 performance data into Prometheus,
using the Icinga 2 Event Stream on one end
and a Prometheus Remote Writer on the other.


Disclaimer


This software is in a proof-of-concept state.
Use at your own risk!


This is not an official Icinga project, just a demo to showcase the Event Stream API and Icinga 2 interoperability in general, although the author is an Icinga employee.


Getting Started


Icinga 2


First, have the Icinga 2 API configured and running.
If using Icinga DB or other modern components, this is already the case.


In your Icinga 2 configuration, add a new ApiUser allowed to query the Event Stream API and reload or restart Icinga 2 afterwards.


object ApiUser "prom" {
  password = "insecure"  # CHANGE ME :>
  permissions = [ "events/checkresult" ]
}


Prometheus


The Prometheus daemon will receive the data via remote write and will not query some exporter.
This decision was made since lots of metrics will be generated and otherwise caching would become an issue...


Thus, start prometheus with the --web.enable-remote-write-receiver command line argument flag.


If the Prometheus web interface is not just accessible from the localhost or third parties may access it,
please configure basic auth.


This Tool


After cloning this repository, start by compiling the application.
To do so, a recent Go compiler is required.


$ go build
$ ./icinga-prometheus-remote-writer -h
2025/03/31 15:07:28 Icinga Prometheus Remote Writer, version g891e95a
Usage of ./icinga-prometheus-remote-writer:
  -eventstream-base-url string
        Icinga 2 API base URL (default "https://localhost:5665/")
  -eventstream-ca-file string
        File containing Icinga 2 CA PEM
  -eventstream-cn string
        Overwrite Server TLS Common Name
  -eventstream-pass-file string
        File containing Icinga 2 API User password
  -eventstream-tls-insecure
        Disable every TLS verification
  -eventstream-user string
        Icinga 2 API User
  -prometheus-base-url string
        Prometheus API base URL (default "http://localhost:9090/api")


Based on the listed command line arguments above, the tool can be instructed.
The following example would forward metrics from a local Icinga 2 to a local Prometheus.


$ ./icinga-prometheus-remote-writer \
  -eventstream-ca-file /var/lib/icinga2/ca/ca.crt \
  -eventstream-cn icinga.example.com \
  -eventstream-user prom \
  -eventstream-pass-file prom-pass
2025/03/31 15:06:47 Icinga Prometheus Remote Writer, version g891e95a
2025/03/31 15:06:47 Connected to Event Stream, forwarding metrics..
[ . . . ]


Metrics


The Icinga 2 CheckResults are converted to three Prometheus metrics:


    

  • icinga_check_result_exit_status contains the exit status of the check command.
    • 

  • icinga_check_result_state contains the numeric Icinga state 0, 1, 2, 3 for OK, WARNING, CRITICAL, UNKNOWN or 0, 1 for UP, DOWN.
    • 

  • icinga_check_result_perf contains performance data entries.
    • 



Each metric contains the following labels:


    

  • object_type is either host or service, depending on the check.
    • 

  • host is the host name.
    • 

  • service is the service name if the check is a service check, otherwise not set.
    • 



The icinga_check_result_perf metric contains the additional labels as well:


    

  • label is the performance data label.
    • 

  • unit is the performance data unit.
    • 



In pratice, the following query would show the RTA of all ping4 and ping6 service checks:


icinga_check_result_perf{service=~"ping(4|6)", label="rta"}


Screenshot of the Prometheus metric plot


Licenses


The code is licensed under the GNU GPL version 2 or later.


Since there are artifacts imported under other licenses, a SPDX header explains the license for each file.
You can use reuse to query all licensing information.