check_cdu

                            
Monitor Server Technology Cabinet Distribution (CDU) Products

                                

        
        

            
            

                

    

    
This is a partial dump of perldoc check_cdu.pl:


Application Requirements


Several standard Perl libraries are required for this program to function. Namely, Net::SNMP,
Getopt::Std, Getopt::Long, Nagios::Plugin::Threshold


General Usage


check_cdu.pl -H  -C  [-t SNMP timeout] [-p SNMP port]


Required Arguments


Only the hostname and community are required. Timeout will default to 2 seconds, port 161.


Thresholds


I opted to use the Nagios::Plugin::Threshold class to handle thresholds. In general I do
not prefer Nagios::Plugins objects, but I just simply could not avoid using the Threshold
class. I apologize for the added dependency, I just could not afford re-inventing the wheel.
The benefit is that the threshold logic used in this plugin follows the standard used in
many other plugins. For reference, here are the general threshold guidelines:


Range definition Generate an alert if x...
10 10, (outside the range of {0 .. 10})
10: ~:10 > 10, (outside the range of {-? .. 10})
10:20 20, (outside the range of {10 .. 20})
@10:20? 10 and ? 20, (inside the range of {10 .. 20})
10 10, (outside the range of {0 .. 10})


Read: http://nagiosplug.sourceforge.net/developer-guidelines.html#THRESHOLDFORMAT
for the full, official, documentation.


Full Documentation


check_cdu is intended to provide extremely flexible and extensive monitoring support for
Server Technology Cabinet Distribution Units (CDU). In general the workflow for this application
follows this procedure:


    

  1. 

    Pull in an entire SNMP table using a Net::SNMP session and get_table().
    

    2. 

  2. 

    Renumerate these "flat" values into a structured hash
    

    3. 

  3. 

    Evaluate any options or thresholds passed on the command line by the user.
    

    4. 

  4. 

    Process the command line options against the data collected from the CDU
    

    5. 

  5. 

    Exit appropriately given the status results
    

    6. 



This workflow is generally followed in four slightly different ways depending on the desired options.


These four procedures are:


    

  1. 

    General System
    

    2. 

  2. 

    Environment
    

    3. 

  3. 

    Towers
    

    4. 

  4. 

    Infeeds
    

    5. 



Environment


An optional feature of a CDU are temperature and humidity probes. On most units, only two T/H
ports exist. When using an EMCU 1-1B in conjuction with a CDU this can be expanded to 4 T/H
ports. I am not aware of a way to increase the number of T/H probes past this amount on a single
CDU. Regardless, this application is designed to support any number of T/H probes.


Prior to running any checks for temperature or humidity, this plugin will check the T/H probe
status. The following states will result in an UNKNOWN return:


    

  • notFound
    • 

  • readError
    • 

  • lost
    • 

  • noComm
    • 



This applies to both temperature and humidity. The LowThresh and HighThresh states are ignored.
Any of these states will issue an UNKNOWN return. Since there is no data available, it's not
logical to initiate a WARNING or CRITICAL and roll someone out of bed. This behavior can easily
be changed in the code, if desired.


In its simplest form, the environment checks will query all available T/H probes connected to the
system. The CDU has internal High/Low thresholds configured for both Temperature and Humidity, and
this is done on a per sensor basis. Without any arguments, this plugin will honor those values.


Considering that there is only one high:low range, I opted to designate this as a WARNING threshold.
This behavior can easily be changed in the code to the CRITICAL state, if desired, but it is NOT
modifiable from the command line. A basic invocation would resemble:


$ check_cdu.pl -H 192.168.0.1 -C public --temp --humid
OK -BLDG_ROOM_RACK, Bottom-Rack-Inlet_F31(A1): 18C, Bottom-Rack-Inlet_F31(A1): 43%,
Bottom-Rack-Exhaust_F32(A2): 33C, Bottom-Rack-Exhaust_F32(A2): 16%, Top-Rack-Inlet_F31(B1): 24C, 
Top-Rack-Inlet_F31(B1): 28%, Top-Rack-Exhaust_F32(B2): 36.5C, Top-Rack-Exhaust_F32(B2): 12%


The plugin output always includes the systemLocation defined on the CDU first. The various objects
queried are then returned in a comma separated list. For temperature and humidity probes, the sensor
Name is returned along with the ID in parantheses. If names haven't been set, the defaults will still
be displayed. Finally the value is listed for each sensor. The temperature scale is automatically
determined from the TempScale object provided via SNMP. For instances where the CDU is configured
for one scale, but the user desires the plugin to report in another scale, the --fahrenheit and
--celsius options are quite handy:


$ check_cdu.pl -H 192.168.0.1 -C public --temp --humid --fahrenheit
OK - BLDG_ROOM_RACK, Bottom-Rack-Inlet_F31(A1): 62.6F, Bottom-Rack-Inlet_F31(A1): 45%, Bottom-Rack-Exhaust_F32(A2): 91.4F


--celsius works in a similar fashion. If a scale is passed to the plugin and the T/H probe is already
configured for that scale, no error will occur. The values will be reported in the native scale for
that sensor.


Expanding on this basic functionality is the --ths option. --ths allows the user to select
which sensors to query, based on the sensor ID (not the name!). --ths will automatically determine
if the sensors exist, and exit UNKNOWN if they were not found. All of the regular sensor status
checks are still performed.


$ check_cdu.pl -H 192.168.0.1 -C public --temp --ths A1,B2 --fahrenheit
OK - BLDG_ROOM_RACK, Bottom-Rack-Inlet_F31(A1): 62.6F, Top-Rack-Exhaust_F32(B2): 97.7F


Note I also left out the --humid option. Either option can be specified alone, or both together,
providing maximum flexibility for designing purpose-built nagios service checks.


User supplied WARNING and CRITICAL thresholds can be applied to the temperature and humidity
sensors using the --warning and --critical directives. This overrides the automatic threshold
logic that relies upon the internal CDU configuration. Either --warning or --critical can be used,
or both can be used together. When querying multiple temperature sensors, a single threshold is
applied across all sensors. The same is true for querying multiple humidity sensors. Both temperature
and humidity can be queried together in the same command, by "chaining" the thresholds together.
Here are a couple examples:


$ check_cdu.pl -H 192.168.0.1 -C public --temp --fahrenheit --ths A1,B1 --warning 60:80
OK - BLDG_ROOM_RACK, Bottom-Rack-Inlet_F31(A1): 62.6F, Top-Rack-Inlet_F31(B1): 77F


(Query just the temperature from T/H probes A1 and B1 and apply a warning threshold to alarm if
either sensor falls below 60F or above 80F)


$ check_cdu.pl -H 192.168.0.1 -C public --humid --ths A2,B2 --warning 10:70
OK - BLDG_ROOM_RACK, Bottom-Rack-Exhaust_F32(A2): 18%, Top-Rack-Exhaust_F32(B2): 13%


(Query just the humidity from T/H probes A2 and B2 and apply a warning threshold to alarm if
either sensor falls below 10% or above 70% relative humidity)


$ check_cdu.pl -H 192.168.0.1 -C public --temp --humid --fahrenheit --ths A1 --warning 80,20: --critical 95,10:
OK - BLDG_ROOM_RACK, Bottom-Rack-Inlet_F31(A1): 64.4F, Bottom-Rack-Inlet_F31(A1): 48%


(Check just sensor A1, but query both temperature and humidity from this sensor. If the temperature
rises above 80F or the humidity falls below 20% generate a WARNING. If the temperature rises above
95 or the humidity falls below 10% generate a CRITICAL.)


IMPORTANT NOTE: When specifying both --temp and --humid the thresholds are chained together as
temperature_threshold,humidity_threshold regardless of which order --temp and --humid are passed!!
aka the following are equivalent:
'--temp --humid --warning 45,60' , '--humid --temp --warning 45,60'
The following are NOT equivalent:
'--temp --humid --warning 45,60', '--humid --temp --warning 60,45'


Towers


Tower state and statistics are checked using the --tower directive. If specified with no arguments
only the overall state of the tower(s) are checked. The ability to query a specific tower does not
exist at this time. If the 'noComm' state is encountered for a tower a WARNING state is generated.
This is likely only possible on a slave tower. If the master tower is in state 'noComm', I doubt you'd
get this far with it ;) If 'fanFail' or 'overTemp' states are encountered, the state is returned as
CRITICAL.


Various metrics from the tower can also be queried by passing them to the --tower directive as a
comma separated list. At the time of development, these metrics are only supported on PIPS units.
A regular SMART or SWITCHED CDU will likely not benefit from any of these enhancements. The plugin
will correctly identify the absence of these metrics if you attempt to query them. The metrics are:


    

  • VACapacity
    • 

  • ApparentPower
    • 

  • VACapacityUsed
    • 

  • ActivePower
    • 

  • Energy
    • 

  • LineFrequency
    • 



It is very important to note that the 'Status' checks are largely skipped when querying any of these
metrics. The 'fanFail' and 'overTemp' states are completely ignored. If the 'noComm' state is
encountered, the metric(s) are skipped and a state UNKNOWN is returned. Given this, to fully utilize
the features of this plugin one should ALWAYS have a service check using just '--tower'. It was not
logical to exit on WARNING/CRITICAL for a 'noComm' state multiple times (say, for instance if there
are separate service checks defined for every metric listed above).


The towers are identified similar to the T/H probes, in the form of NAME(ID): VALUE. These are all
configurable on the CDU itself. Typically, a circuit name would be used for a Tower name. Thresholds
are applied in a similar manner to the --temp and --humid checks. ORDER DOES MATTER. The order in
which the metrics are listed is the order in which the thresholds should be "chained". The same logic
applies to these thresholds, see the THRESHOLDS section for specifics.


Here are some examples:


$ check_cdu.pl -H 192.168.0.1 -C public --tower
OK - BLDG_ROOM_RACK, TowerA(A) Status: normal(0), TowerB(B) Status: normal(0)a

$ check_cdu.pl -H 192.168.0.1 -C public --tower ApparentPower,ActivePower,VACapacityUsed --warning 1200,1000,30
OK - BLDG_ROOM_RACK, TowerA(A) ApparentPower: 993VA, TowerA(A) ActivePower: 939W, TowerA(A) VACapacityUsed: 9.1%,


TowerB(B) ApparentPower: 927VA, TowerB(B) ActivePower: 870W, TowerB(B) VACapacityUsed: 8.5%


(Check that ApparentPower does not exceed 1200VA, ActivePower does not exceed 1000W and the Capacity
used does not exceed 30%. If any of these scenarios occur, generate a WARNING)


$ check_cdu.pl -H 192.168.0.1 -C public --tower Energy --warning 10000 --critical 15000
OK - BLDG_ROOM_RACK, TowerA(A) Energy: 6654kWh, TowerB(B) Energy: 7658kWh


(If the kWh consumption of either tower exceeds 10,000 generate a WARNING. If it exceeds 15,000
generate a CRITICAL. Say you're in a co-lo paying for power utilization and your piggy bank will
run dry if you use too much power ...)


$ check_cdu.pl -H 192.168.0.1 -C public --tower VACapacity --warning 10800
WARNING - BLDG_ROOM_RACK, TowerA(A) VACapacity: -1VA


(This is a very bizarre but interesting scenario. I included VACapacity because it was there, but
who would logically check a static value such as the capacity of a tower? Well, it turns out that
this particular unit is slightly broken and the Capacity is -1. This should just provide some ideas
on why it may be useful to monitor things that otherwise wouldn't make sense)


Infeeds


Infeed state and statistics are checked using the --infeed directive. It is very similar to the --tower
check. If specified with no agruments, the infeed 'Status' and 'LoadStatus' objects are checked. The
ability to query a specific infeed does not exist at this time (and likely never will). The following
infeed Statuses will generate a WARNING:


    

  • noComm
    • 

  • offWait
    • 

  • onWait
    • 

  • off
    • 



A CRITICAL will be generated if the Infeed has the following Status:


    

  • offError
    • 

  • onError
    • 



Likewise the LoadStatus object is checked for each infeed as well. A WARNING is generated for the
following LoadStatus conditions:


    

  • noComm
    • 

  • reading
    • 

  • loadLow
    • 



I wasn't sure what the 'reading' state was, this state is also present across many other CDU
objects. There is a good chance this state simply infers that the state is currently being
"read" or updated, and it's likely that this state will be ignored in future versions of the
plugin if that is the case. The loadLow must be determined by an internal CDU threshold, however
this threshold isn't available via SNMP - so I left it alone. A CRITICAL is generated for the
other LoadStatus states:


    

  • notOn
    • 

  • loadHigh
    • 

  • overLoad
    • 

  • readError
    • 



Simple modifications to the code can be done to move these various Statuses between the CRITICAL
and WARNING states if desired, but it is not possible from the command line.


Similar to the --tower directive, many of these Status checks are skipped when querying specifc
metrics from the infeed. If any metrics are provided to --infeed, the infeed Status is checked
for the 'noComm' status. If this is true, the plugin will append this to the UNKNOWN 'bucket'
and skip checking the metric. The following infeed metrics are currently supported:


    

  • PhaseVoltage *
    • 

  • Voltage
    • 

  • CapacityUsed *
    • 

  • Power
    • 

  • ApparentPower *
    • 

  • Energy *
    • 

  • LoadValue
    • 

  • PhaseCurrent *
    • 



* These metrics are only available on PIPS units.


The infeeds are identified similar to the T/H probes, in the form of NAME(ID): VALUE. These are all
configurable on the CDU itself. Typically, a circuit name would be used for an infeed name. Thresholds
are applied in a similar manner to the --temp and --humid checks. ORDER DOES MATTER. The order in
which the metrics are listed is the order in which the thresholds should be "chained". The same logic
applies to these thresholds, see the THRESHOLDS section for specifics.


Some examples:


$ check_cdu.pl -H 192.168.0.1 -C public --infeed
OK - BLDG_ROOM_RACK, TowerA_InfeedA(AA) Status: on(1), TowerA_InfeedA(AA) LoadStatus: normal(0), 
TowerA_InfeedB(AB) Status: on(1), TowerA_InfeedB(AB) LoadStatus: normal(0), TowerA_InfeedC(AC) Status:
on(1), TowerA_InfeedC(AC) LoadStatus: normal(0), TowerB_InfeedA(BA) Status: on(1), TowerB_InfeedA(BA)
LoadStatus: normal(0), TowerB_InfeedB(BB) Status: on(1), TowerB_InfeedB(BB) LoadStatus: normal(0),
TowerB_InfeedC(BC) Status: on(1), TowerB_InfeedC(BC) LoadStatus: normal(0)


(This is a basic tower check for a master/slave 3 phase CDU. There are 6 infeeds total across both
towers, and two separate checks are performed (Status,LoadStatus) for each infeed. This is a lot of data)


$ check_cdu.pl -H 192.168.0.1 -C public --infeed LoadValue --warning 12 --critical 24
OK - BLDG_ROOM_RACK, TowerA_InfeedA(AA) LoadValue: 4.07A, TowerA_InfeedB(AB) LoadValue: 3.21A,
TowerA_InfeedC(AC) LoadValue: 1.62A, TowerB_InfeedA(BA) LoadValue: 3.61A, TowerB_InfeedB(BB) LoadValue:
2.76A, TowerB_InfeedC(BC) LoadValue: 1.73A


(This is a simple load/current check which applies a warning and critical threshold to the load of all 6
infeeds on a dual tower 3 phase CDU.)


$ check_cdu.pl -H 192.168.0.1 -C public --infeed ApparentPower,CapacityUsed --warning 1000,20
OK - BLDG_ROOM_RACK, TowerA_InfeedA(AA) ApparentPower: 673VA, TowerA_InfeedA(AA) CapacityUsed: 12.6%,
TowerA_InfeedB(AB) ApparentPower: 0VA, TowerA_InfeedB(AB) CapacityUsed: 10.5%, TowerA_InfeedC(AC)
ApparentPower: 317VA, TowerA_InfeedC(AC) CapacityUsed: 5.3%, TowerB_InfeedA(BA) ApparentPower: 575VA,
TowerB_InfeedA(BA) CapacityUsed: 12%, TowerB_InfeedB(BB) ApparentPower: 0VA, TowerB_InfeedB(BB) CapacityUsed:
8.9%, TowerB_InfeedC(BC) ApparentPower: 348VA, TowerB_InfeedC(BC) CapacityUsed: 5.7%


(Generate a warning if the ApparentPower of any infeed exceeds 1000VA, and generate a warning if the
Capacity Used exceeds 20% on any infeed)


PhaseVoltage and PhaseCurrent use the PhaseID instead of infeedID in the plugin output. Throughout our
testing, it has been difficult to ascertain a difference between PhaseVoltage and Voltage. There is
generally a considerable difference between PhaseCurrent and LoadValue, however it most likely makes
sense to only check one of these.


Enhanced Infeed checks


There are two additional metrics that can be checked with the '--infeed' directive. They are:


    

  • LoadImbalance
    • 

  • 

    VoltageImbalance
    

    These metrics are not provided directly by the CDU, rather they are computed internally by the plugin.
    

    • 



Please note, these special metrics are ONLY available on 3 phase units. Some versions of the CDU
firmware provide a '3-Phase Load Out-of-Balance Threshold' setting and the results are displayed on
the 'istat' menu. None of this information is provided via SNMP. Thresholds are required for either
of these computed metrics. Unlike the display in 'istat' only the out-of-balance infeed(s) will be
displayed, not infeeds across the entire tower. I used a basic 3 phase motor load phase imbalance
equation to generate the imbalance percentages for both Current and Voltage:


Percent imbalance = maximum deviation from average / average of three phases * 100


When an infeed is queried for either voltage or current imbalance, the plugin determines which tower
the infeed is a part of. All infeed values (voltage or current) for that tower are then averaged
together. The deviation from the average is then determined for this particular infeed, accomodating
either a negative or positive delta from the average. This is then divided by the average and
multiplied by 100 to determine the percent imbalance. This equation was pulled from the following
document:


http://support.fluke.com/educators/download/asset/2161031\_b\_w.pdf


An example invocation of this check would look like:


$ check_cdu.pl -H 192.168.0.1 -C public --infeed LoadImbalance --warning 20 --critical 30
CRITICAL - BLDG_ROOM_RACK, TowerA_InfeedA(AA) LoadImbalance: 39.07%, TowerA_InfeedC(AC)
LoadImbalance: 50.46%, TowerB_InfeedA(BA) LoadImbalance: 33.54%, TowerB_InfeedC(BC) LoadImbalance: 34.54%


(Generate a WARNING if the load imbalance of any infeed exceeds 20%, and a CRITICAL if the imbalance
exceeds 30%. Clearly, this is not a well balanced rack! Hence the need for such a check)


The same can be done for voltage, however the margins should be much, much smaller than load.


This can be useful to detect bad incoming power conditions. Unfortunately this only evaluates
an imbalance across the phases of a single tower. A more useful approach would be to judge
imbalance between two separate towers, and hence two separate feeds/circuits which could be
coming from two separate sources (ie. UPS/utility). Currently that functionality does not exist.


Here is an example:


$ check_cdu.pl -H 192.168.0.1 -C public --infeed VoltageImbalance --warning .5 --critical 2
WARNING - BLDG_ROOM_RACK, TowerB_InfeedB(BB) VoltageImbalance: 0.65%


(Generate a WARNING if the imbalance between voltages per infeed is greater than .5% and a
CRITICAL if the imbalance is greater than 2%)


Plugin Termination


Numerous scenarios exist where the plugin will exit abnormally. This could be due to user input error,
or failure to retrieve required SNMP data, etc. In all identifiable cases, the plugin will exit with an
UNKNOWN state and a descriptive message indicating the failure. Users should be aware that if all SNMP
calls fail, monitoring of the CDU may be effectively rendered useless if UNKNOWN states are not reported
(this is common). This is dissimilar to plugins like check_nrpe that exit CRITICAL if an SSL negotiation
error occurs!


Throughout the workflow of the plugin metrics are evaluated against thresholds and the results are placed
into various 'buckets' reflecting OK,WARNING,CRITICAL and UNKNOWN states. At the end of the workflow,
reporting is done based upon the presence or absence of these buckets. If both CRITICAL and WARNING
conditions exist, they are BOTH reported in the plugin_output text, however the state is reported as
CRITICAL. An example of this can be seen in the following output:


$ check_cdu.pl -H 192.168.0.1 -C public --temp --humid --ths A1 --warning 16,30 --critical 20,40
CRITICAL - BLDG_ROOM_RACK, Bottom-Rack-Inlet_F31(A1): 43%, WARNING - Bottom-Rack-Inlet_F31(A1): 17C


Some options end up producing a large amount of output, and this could easily exceed what Nagios can
accept, or also exceed character limits on various notification devices (maybe you're tweeting your
CDU status for instance ;P) The '--oksummary' option exists to summarize the output for any type of
check being done. If all metrics being checked are in state 'OK' the output supresses the specifics
of these metrics and simply reports 'N metrics are OK' The version and location are also displayed
in the plugin_output.