Unless stated otherwise, all the following commands should be run as root.
Steps which are not mentioned in the official documentation are shown in italics. Steps in the official documentation which you should not perform are also shown in italics.
Icinga2 supports the concept of a Master server which is (almost) the single point of configuration for an entire network of monitored systems, which communicates either with each of those systems (Agents) directly, or which communicates with one or more Satellite servers, which then each communicate with some of the Agents. Every Agent machine communicates only with one Master or Satellite, and agents can be grouped with Satellites in any way which makes geographical, logical or organisational sense.
This documentation assumes that you do have all three levels; if you do not use a Satellite, then the Agents are simply connected directly to the Master, so these instructions work just as well for that scenario - simply skip anything to do with Satellite configuration.
It is also possible to have a four-layer setup with two levels of Satellites between the Master and the Agents. This is described further in the instructions below. I do not yet know whether four is the limit…
Icinga2 has been developed to support all three layers running the same version of Icinga2 (of course), and also to support each layer being one major version lower than the layer above (this is assuming that "Icinga2" is the name of the application, and "Icinga2.10.6" is major version 10, minor version 6).
So, for example, you can do: Master 2.11.x ⇒ Satellite 2.10.x ⇒ Agent 2.9.x
Larger variations between layer versioning may work but are not officially supported.
I don't know what the rule is if you manage to make a four-layer (or more?) setup work.
Based on https://icinga.com/docs/icinga-2/latest/doc/06-distributed-monitoring/#master-setup
It is assumed that you have already completed the basic setup on this machine.
Run the node wizard or the node setup command (see details below) once on the Master, before setting up any Satellites or Agents:
icinga2 node wizard
icinga2 node setup --parameters...
I'm assuming you will use "On-Demand CSR Signing", since this is a good deal simpler than "CSR Auto-signing".
It is also assumed that you have already completed the basic setup on this machine.
The same two methods can be used to set up a Satellite machine - the "wizard" which prompts you for answers to questions; or supplying all the answers as parameters to a command which then achieves the same thing - see below.
Obviously, the latter is ideal for any sort of automated setup (puppet, chef, ansible, DIY scripts, etc).
If you want two layers of Satellites between the Master and the Agents, simply repeat the instructions for the second (lower) level of Satellite, entering the details of its Parent Satellite instead of the Master.
I have not yet tried three Satellite layers, and I find it slightly hard to think of a plausible use case for this.
It is assumed that you have already completed the basic setup on this machine.
This is almost identical to the Satellite setup described above, except that the parent node is now the Satellite to which this Agent should connect, rather than the Master. All information in the setup of an Agent should relate to the Agent or its direct Satellite, not to the Master.
The final step, of signing the certificate request, is still performed on the Master, however.
You can use either the wizard or the parameter method of setting up the Agents - see below.
The node setup performs all necessary configuration on the Agent node, however it does not add the Agent node to the upstream configuration (Master and Satellite). The Master needs to know about the Host and what Service Checks to perform on it, and the Satellite needs to know about the Zone and the Endpoint in order to communicate with it.
On the Master:
icinga2 node wizard
/etc/init.d/icinga2 restart
On Satellites or Agents:
icinga2 node wizard
You can find out what parameters can be supplied using the command:
icinga2 node setup --help
Setting up the node in this way is a two-step process:
The first step is accomplished with:
mkdir /var/lib/icinga2/certs chown nagios: /var/lib/icinga2/certs icinga2 pki save-cert --trustedcert /var/lib/icinga2/certs/ca.crt --host FQDN
Replace "FQDN" with the address or DNS hostname of the parent node (Master or Satellite).
It surprises me that you have to create the certificate directory yourself, rather than the pki command doing it for you, but no doubt there's a good reason for it somewhere. If you use the wizard command, it does create the directory.
The second step is accomplished with:
icinga2 node setup --cn Thisnode --zone Thisnode --trustedcert /var/lib/icinga2/certs/ca.crt --parent_host Parent.FQDN --endpoint Parent.CN --parent_zone Parent.CN --accept-config --accept-commands --disable-confd
Replace "Thisnode" with the CN you wish to use for this node, "Parent.FQDN" with the IP address or DNS hostname of the parent machine, and "Parent.CN" with the CN you assigned to the parent.
icinga2 ca list icinga2 ca sign xxxxx
/etc/init.d/icinga2 restart
Icinga has a very strange definition of "Zone" - it is not a geographical region, and it is not a notional collection of machines. It is simply something which needs to be defined for each machine in the system (Masters, Satellites and Agents) and contains only that single machine (there is a small exception if you set up High Availability for Icinga, where you can place two Masters, or two Satellites, into a single zone, but two is the limit). I have no idea why the developers thought this concept was useful, or why they chose the name "Zone" for it. I find it especially misleading that they even use "Europe", "USA" and "Asia" as example zone names in the documentation.
The end result is that for each machine to be monitored, you need three definitions: Zone, Endpoint, and Host, for example:
object Zone "Website" { endpoints = [ "Website" ] }
object Endpoint "Website" { host = "Website" }
object Host "Website" { address = "www.example.com" }
There's a strict 1:1:1 correspondence (with a small exception for HA): a Zone contains one Endpoint (and an Endpoint exists in one Zone); an Endpoint contains one Host, and a Host is in one Endpoint.
The single exception for High Availability is that a Zone may contain two Endpoints (but no more). This only makes sense for Masters or Satellites, and it means that the two machines share the work expected of that Zone.
This combination of Zones, Endpoints and Hosts does seem unnecessarily complicated to me.
There is also an absolutely crucial correspondence between the hierarchy of your network and the directory structure you create under /etc/icinga2/zones.d on the Master node, and (in my opinion) this is not well explained in the official documentation. I managed to find a helpful article by one of the Icinga developers which explains this better, but does not pretend to be a step-by-step guide to Getting it Right.
Under /etc/icinga2/zones.d on the Master server, you should have the following:
If you want to have two layers of Satellites, then the above directory structure becomes:
Here is an example, based on a master named Eyeball, a Satellite named Middleman and two Agents connected to the Satellite called Fred and George:
The Zone definitions for each top-level Satellite specify the Master as their parent node, and the Zone definitions for each Agent or lower-level Satellite specify the corresponding Satellite as their parent node.
In addition to this, every machine also needs Zone and Endpoint definitions for itself and its directly-connected parent in /etc/icinga2/zones.conf. On Satellite and Agent machines, this is the only file which should contain configuration data (the rest is propagated down to the machine from the Master), and this file should automatically and correctly be created for you by the node wizard or node setup commands.
Icinga2 supports two methods by which the Master tells the Satellites and the Agents what Service Checks to perform and when. There used to be three, but the one called Bottom-up was deprecated in version 2.6 (2016) and eliminated in 2.8 (2017).
The two methods now available are:
I see no reason to use Top-Down Command Endpoint - it places more load on the Master, and if the connection between Master and Satellite, or between Satellite and Agent, breaks, no checks will be performed.
So, I would always use Top-Down Config Sync, and the instructions above for setting up the Satellites and Agents are based on that mechanism.
Go up
Return to main index.