A script to gather valuable data for providing to support engineer
This script is mostly intended as tool for support engineers. If a user reports a problem, the output of the scripts should cover the most common questions a support engineer will have.
The intention is not to collect as much data as possible but collect what is helpful in most cases and providing an overview whithout drowning support in too much data.
Please note: Even though this script is meant to just collect really useful data, it does still collect a lot. Browse through any output before you send it so that you don't send any confidential data without noticing.
Please note as well: The main intention of the script is to provide data when asked for it. While it might be helpful when working with commercial support, sending unwanted big blobs of data into community channels might lower your chances of getting help. Always try the troubleshooting section of the Icinga 2 documentation before requesting any community help. The upcoming "suggestion mode" of this script might help with troubleshooting as well.
Later versions might include options to collect this "usually helpful" set of data or lots of data for deep diving (e.g. whole logfiles, etc.)
This script has several different modes:
- Default mode: Give a quick overview over your setup for professional or community support
- Tarball mode: Create a tarball of the configuration of your setup for thorough investigation by professional support
- Suggestion mode: Search things to improve and possible problems and give advice (still to be implemented)
If your setup consists of multiple hosts you can run on it on every node to get a more thorough overview. If you have e.g. dedicated nodes for graphing the script will work and only analyse the tools you installed on the node. (Although details about Graphers are still missing)
Just use the script without any parameter.
Just run the script with the
# ./icinga-diagnostics.sh -t
Be aware that this collects your whole configuration including passwords, etc. Think before you send this to someone!
The script will thell you where you can find your tarball when it's finished. You can still remove data from the tarball before sending it to someone.
- -z : Calculate zones with their endpoints (very time consuming on large setups)
- -g : add
gdb Output is only useful for debugging problems in the Icinga 2 binary, not for configuration problems. For
gdb to work you have to install
gdb and the "degbugging symbols" for all the programs you want to have their information added. e.g. for Icinga 2 the corresponding package is
icinga2-debuginfo on RedHat/CentOS or
icinga2-dbg on Debian or Ubuntu. In addition you will need the
icinga-gdb file containing the
gdb commands in the directory where icinga2-diagnostics resides.
Icinga 2 configuration
In the folder
debug-configuration you find a sample configuration file you can place in a zone where you have problems with executing checks with Icinga. This will create some services that check for the presence of some script interpreters including version and include paths on the host. You will get extra services showing you the user and other data about this specific host.
This is especially helpful if you have problems with finding the reason why a specific check is not able to be executed. A common situation is that you can run a check as
icinga user via shell but the Icinga daemon can't.
This configuration is not intended to be running all the time. Just copy it into to zone you want to debug and when you're finished, remove it from Icinga.
Currently supported systems
The script currently works on the following operating systems:
- RHEL/CentOS/Oracle Linux/Scientific Linux
- SLES (thanks, wnieder)
- FreeBSD (thanks, @larsengels)
- Debian (still some functionality is missing)
The script currently works with the following software components:
- Icinga 2
- Icinga Web 2
Planned supported systems
The script should work on all Operating systems supported by Icinga 2
The script should gather informations about the most common addons to Icinga 2 and collect data relevant to monitoring setups.
This is an abrreviated output from a run on a Icinga 2 Vagrant box
# ./icinga-diagnostics.sh ### Icinga 2 Diagnostics ### # Version: 0.0 # Run on icinga2 at Thu Jan 11 18:02:14 UTC 2018 Running as root ## OS ## OS Version: CentOS Linux release 7.4.1708 (Core) Hypervisor: Running virtually on a VirtualBox hypervisor CPU cores: 2 RAM: 1.8G SELinux: Permissive Firewall: inactive # Icinga 2 # ## Packages: ## Icinga 2 Version : 126.96.36.199.g025abc3 Done checking packages. See Anomaly section if something odd was found. Features: Disabled features: command compatlog elasticsearch gelf influxdb livestatus opentsdb perfdata statusdata syslog Enabled features: api checker debuglog graphite ido-mysql mainlog notification Check intervals: 88 * check_interval = 300, Host 1 * check_interval = 60, Host 113 * check_interval = 60, Service 13 * check_interval = 30, Service 5 * check_interval = 5, Service 3 * check_interval = 300, Service Used commands (numbers are relative to each other, not showing configured objects): 28093 /etc/icinga2/scripts/mail-service-notification.sh 14120 /usr/lib64/nagios/plugins/check_mysql_health 10146 /usr/lib64/nagios/plugins/check_http 6428 /usr/lib64/nagios/plugins/check_ping 4396 /usr/lib64/nagios/plugins/check_load 3415 /usr/lib64/nagios/plugins/check_ssh 1758 /usr/lib64/nagios/plugins/check_dns 1758 /usr/lib64/nagios/plugins/check_disk 880 /usr/lib64/nagios/plugins/check_procs 879 /usr/lib64/nagios/plugins/check_users 879 /usr/lib64/nagios/plugins/check_swap 528 /usr/bin/icingacli information/cli: Icinga application loader (version: v2.8.0-187-g025abc3) information/cli: Loading configuration file(s). information/ConfigItem: Committing config item(s). information/ApiListener: My API identity: icinga2 warning/globals.getHostGeoLocation: Cannot find 'be' in GeoLocationShort warning/ApplyRule: Apply rule 'many-dummy' (in /etc/icinga2/demo/many.conf: 36:1-36:39) for type 'Notification' does not match anywhere! warning/ApplyRule: Apply rule 'many-dummy' (in /etc/icinga2/demo/many.conf: 41:1-41:42) for type 'Notification' does not match anywhere! warning/ApplyRule: Apply rule 'many-test-0' (in /etc/icinga2/demo/many.conf: 16:3-16:32) for type 'Service' does not match anywhere! warning/ApplyRule: Apply rule 'many-test-1' (in /etc/icinga2/demo/many.conf: 16:3-16:32) for type 'Service' does not match anywhere! warning/ApplyRule: Apply rule 'many-test-2' (in /etc/icinga2/demo/many.conf: 16:3-16:32) for type 'Service' does not match anywhere! information/ConfigItem: Instantiated 1 ApiListener. information/ConfigItem: Instantiated 3 Zones. information/ConfigItem: Instantiated 1 Endpoint. information/ConfigItem: Instantiated 4 ApiUsers. information/ConfigItem: Instantiated 2 FileLoggers. information/ConfigItem: Instantiated 28 Notifications. information/ConfigItem: Instantiated 1 UserGroup. information/ConfigItem: Instantiated 2 Users. information/ConfigItem: Instantiated 203 CheckCommands. information/ConfigItem: Instantiated 3 NotificationCommands. information/ConfigItem: Instantiated 4 HostGroups. information/ConfigItem: Instantiated 1 IcingaApplication. information/ConfigItem: Instantiated 89 Hosts. information/ConfigItem: Instantiated 3 TimePeriods. information/ConfigItem: Instantiated 134 Services. information/ConfigItem: Instantiated 3 ServiceGroups. information/ConfigItem: Instantiated 1 CheckerComponent. information/ConfigItem: Instantiated 1 GraphiteWriter. information/ConfigItem: Instantiated 1 IdoMysqlConnection. information/ConfigItem: Instantiated 1 NotificationComponent. information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars' information/cli: Finished validating the configuration file(s). # Icinga Web 2 # Packages: icingaweb2-188.8.131.52.g890013c-0.2017.11.28+2.el7.icinga.noarch package php is not installed httpd-2.4.6-67.el7.centos.6.x86_64 Icinga Web 2 Modules: MODULE VERSION STATE DESCRIPTION businessprocess 2.1.0 enabled A Business Process viewer and modeler cube 1.0.1 enabled Cube for Icinga Web 2 director 1.3.2 enabled Director - Config tool for Icinga 2 doc 2.5.0 enabled Documentation module grafana 1.1.10 enabled Grafana - A perfdata visualisation module map 1.0.4 enabled Map - Visualize your hosts and service status monitoring 2.5.0 enabled Icinga monitoring module businessprocess via git - "00e2f1886a9b07244e8dad237776b629fad59c0a" cube via git - "7ba3feb71601fd2433e0b787ea87dddf53878e49" director via git - "c4a97692df23e428a2eb6f1be41a7c25ea7b19a4" doc via release archive/package grafana via git - "5ba8995a15fb4b1e72232e3dffd4b45c6bb89ab8" map via git - "ec7fed3e4085f98f6f30f3557d85d5ed498e2a7d" monitoring via release archive/package Icinga Web 2 commandtransport configuration: [icinga 2 api localhost] transport = "api" host = "localhost" port = "5665" username = "icingaweb2" password = MASKED Director is release 1.3.2 Director was installed as a git clone # Anomalies found # * Director is installed but no release archive was used for installation. (Please note that it still could the code of a release)