Icinga Integration

Icinga is an open source IT infrastructure monitoring tool that offers monitoring and alerting for systems, network devices, applications, and services. Icinga is highly configurable, easily extensible, hence very pervasive.

1200

Opsgenie's Icinga plugin supports a bi-directional integration with Icinga. The integration leverages Opsgenie's Icinga-specific executable and Marid utility to automatically create rich alerts (status, alert histogram, trends, etc.) and synchronizes alert status between Icinga and Opsgenie.

Opsgenie Icinga Plugin Integration

Opsgenie Icinga integration plugin utilizes full capabilities of Opsgenie and provides bi-directional integration with Icinga. Integration leverages Opsgenie's Icinga-specific executable and Marid utility to automatically create rich alerts (status, alert histogram, trends, etc.) and synchronizes alert status between Icinga and Opsgenie.

442

Installation

The steps below describe how to integrate Opsgenie and Icinga using Opsgenie Icinga integration plugin. Note that slight alterations may be needed for these instructions depending on your exact Linux distribution and your Icinga configuration.

🚧

If you're using Lamp based Icinga plugin, backup all existing configurations. Uninstall the old plugin, then install the new one.

Prerequisites

Packages provided support the following systems:

  • Red Hat based linux distributions
  • Debian based linux distributions

Download Opsgenie Icinga Plugin

For Red Hat Based Distributions

rpm -i opsgenie-icinga-<your_version>.rpm

🚧

During upgrades, rpm package does not overwrite your existing configurations. It saves the new default configuration file as opsgenie-integration.conf.rpmnew. Find more information about rpm upgrades config file handling from here.

🚧

If you want to update from version 201X-XX-XX to 2.X.X, you must add --force parameter. E.g.:
rpm -U --force opsgenie-integration-<your_version>.rpm

We suggest that you backup existing configuration files before update!

For Debian Based Distributions

dpkg -i opsgenie-icinga-<your_version>.deb

Add Icinga integration to Opsgenie

  1. Please create an Opsgenie account if you haven't done so already.
  2. Go to Opsgenie's Icinga Integration page.

🚧

For Free and Essentials plans, you can only add the integrations from the Team Dashboards, please use the alternative instructions given below to add this integration.

  1. Specify who is notified of Icinga alerts using the Teams field. Auto-complete suggestions are provided as you type.

📘

An alternative for Step 2) and Step 3) is to add the integration from the Team Dashboard of the team which will own the integration. To add an integration directly to a team, navigate to the Team Dashboard and open Integrations tab. Click Add Integration and select the integration that you would like to add.

  1. Copy the API key generated for the integration. This key is used by Icinga to authenticate with Opsgenie and specify the integration to be used to process Icinga alerts.
  2. Click Save Integration.
2151

Opsgenie Plugin Configuration in Icinga

The plugin uses a golang-executable file (included in the plugin as icinga2opsgenie) to create, acknowledge, and close alerts in Opsgenie. Configure Icinga to execute this file on events to create, acknowledge, and close alerts in Opsgenie.

Configuration ParameterDescriptionMandatory to fill
apiKey Copy the API key from the Icinga integration you've created above. icinga2opsgenie uses this key to authenticate to Opsgenie. API key is also used to identify the right integration configuration that should be used to process alerts.Yes
recipients Recipients field is used to specify who should be notified for the Icinga alerts. This field is used to set the default recipients field value. It can be modified to route different alerts to different people in Opsgenie Icinga integration, Advanced Settings page. Recipients can be set to users, groups, escalations or schedules who will be notified by Opsgenie. If you did not set recipients in the integration, this field is required.Optional
teams Teams field is used to specify which teams should be notified for the Icinga alerts. This field is used to set the default teams field value. It can be modified to route different alerts to different teams in Opsgenie Icinga integration, Advanced Settings page.Optional
tags Tags field is used to specify the tags of the alert that created in Opsgenie.Optional
icinga_server icinga_server field is used to identify the Icinga server in Opsgenie, and only required when there are multiple Icinga servers. This field is used by Opsgenie when sending actions executed by users (acknowledge, close, etc.) back to your Icinga servers via MaridOptional
viaMaridUrl viaMaridUrl field is used to send alerts to Opsgenie through Marid. You should enter host and port values of your working Marid.
Useful when Icinga server has no internet connection but Marid has internet connection.
In order to use this feature you should be running the Marid provided within Opsgenie Icinga Plugin
Marid should be running with web server enabled ( http or https configurations enabled )
Marid can run on a seperate host server, the communication between icinga2opsgenie & Marid is done with basic http.
* Helps Icinga server to consume less time when sending data to Opsgenie by letting Marid do the long task with an async approach.
Optional
logPathSpecifies the full path of the log file. (Default value is /var/log/opsgenie/icinga2opsgenie.log)Optional
icinga2opsgenie.http.proxy.enabled icinga2opsgenie.http.proxy.enabled field is to enable/disable external proxy configuration. The default value is false.
Optional
icinga2opsgenie.http.proxy.host It is the host of the proxy.Optional
icinga2opsgenie.http.proxy.port It is the port of the proxy.Optional
icinga2opsgenie.http.proxy.scheme It is the proxy connection protocol. It may be http or https depending on your proxy servers. Its default value is http.Optional
icinga2opsgenie.http.proxy.username It is the Proxy authentication username.Optional
icinga2opsgenie.http.proxy.password It is the Proxy authentication password.Optional
opsgenie.api.urlIf you're using Opsgenie from another domain(eg. EU, sandbox), you should update this configuration.Optional

There are three ways to configure golang-executable file:

  1. Configuring from conf file: Configure from /etc/opsgenie/conf/opsgenie-integration.conf file. Configuring from conf file will overwrite the configurations made in the script.
  2. Configuring by using Golang Flags: Configure by entering flags to command in the opsgenie.cfg file. Use -apiKey flag for your apiKey and -is flag for your icinga_server name. If multiple Icinga servers aren't being used, the Icinga server does not need to be defined. Using flags overwrites all the other configuration methods mentioned above.

📘

To send additional custom arguments, add them after the flags as: customArgName1 customArgValue1 customArgName2 customArgValue2
Parse custom arguments by adding {{_payload.customArgName}} to wherever is needed in the input fields.
For more information about using raw parameters please visit Dynamic Fields document.

  1. Configuring from script: Configure apiKey and icinga_server from icinga2opsgenie.go script. If this option is used, build the script again and put the new executable to /usr/bin directory. Find information about the location of the icinga2opsgenie.go and how to build a go script in the Source section.

📘

Configure the golang-executable to use a proxy for sending HTTP requests by defining the environment variable HTTP_PROXY=http://host:port

Define Icinga contacts

  1. Copy /etc/opsgenie/opsgenie.cfg config file, (configures a contact and its host and service notification commands) to /usr/local/icinga/etc/objects directory.
cp /etc/opsgenie/opsgenie.cfg /usr/local/icinga/etc/objects
  1. Add the following line to the main Icinga configuration file (ICINGA_HOME/etc/icinga.cfg)
...
cfg_file=/usr/local/icinga/etc/objects/opsgenie.cfg
...
  1. Add the contact "opsgenie" to the Icinga configuration’s main contact group in ICINGA_HOME/etc/objects/contacts.cfg file. If using the default configuration, contacts.cfg, add "opsgenie" user to the "admins" contact group.
  2. Restart Icinga.

If everything goes well, alerts are seen in Opsgenie for every notification created in Icinga.

1091

Configure Opsgenie to Icinga Integration (Optional)

🚧

If you are using Opsgenie Edge Connector instead of Marid, you can find the integration specific script and its sample config from here. For more information about OEC, please refer OEC Integration documentation

The plugin uses Marid utility (included in the plugin) to enrich alerts when created and to update the state of the them in Icinga when updated in Opsgenie. For example, when an alert is created in Opsgenie, Marid gets the details (histogram, trends etc.) from Icinga and attaches them to the alert. When users acknowledge an alert from their mobile devices using the Opsgenie app, the alert gets acknowledged in Icinga, and when users add comments to alerts in Opsgenie, comments get posted to Icinga as well. Marid subscribes to alert actions in Opsgenie and reflects these actions on Icinga using Icinga CGIs.

🚧

Marid uses Classic UI CGIs to issue commands on Icinga. To allow Marid to post updates on Icinga, put disable_cmd_cgi_csrf_protection=1 into /etc/icinga/cgi.cfg and restart the Icinga service.

  • To start Marid, run following command:
/etc/init.d/marid start
  • To stop Marid, run following command:
/etc/init.d/marid stop

Marid is a java application; therefore requires the Java Runtime version 1.6+ Both the Open JDK and Oracle JVMs can be used.

📘

In order to use this feature "Send Alert Actions To Icinga" checkbox should be enabled in Opsgenie Icinga Integration.

📘

Ensure that JAVA_HOME environment variable is set. If it is not, you may set it by removing the comment at the begining of the following line in /etc/opsgenie/profile file and set JAVA_HOME to your JRE installation directory.
#JAVA_HOME=<path/to/JDK or JRE/install>

To execute actions in Icinga, Marid gets the configuration parameters from /etc/opsgenie/conf/opsgenie-integration.conf file.

Configuration ParameterDescription
icinga.alert_histogram_image_url Marid retrieves histogram images from Icinga using this URL. Localhost should be replaced with the Icinga server address.
icinga.trends_image_url Marid retrieves trends images from Icinga using this URL. Localhost should be replaced with the Icinga server address.
icinga.command_url URL to update Icinga alerts when alerts get acknowledged, commented, etc.
icinga.user Username to authenticate Icinga web server to get Icinga histogram and trends images.
icinga.password Password to authenticate Icinga web server to get Icinga histogram and trends images.
icinga.http.timeout Timeout duration in msecs to get Icinga histogram and trends images.

Multiple Icinga Server Support

Configure Marid to forward Opsgenie alert actions to multiple Icinga servers:

  • On each Icinga Server, modify /etc/opsgenie/conf/opsgenie-integration.conf icinga_server config property value to a unique name.
  • On Marid server, modify /etc/opsgenie/conf/opsgenie-integration.conf, Add commandurl, user, password configuration for each Icinga server.
  • opsgenie-integration.conf has a commented sample configuration for multiple Icinga servers :
#icinga.server1.alert_histogram_image_url=http://icingaHost:port/icinga/cgi-bin/histogram.cgi
#icinga.server1.trends_image_url=http://icingaHost:port/icinga/cgi-bin/trends.cgi
#icinga.server1.command_url=http://icingaHost:port/icinga/cgi-bin/cmd.cgi
#icinga.server1.user=icingaadmin
#icinga.server1.password=icingaadmin
#icinga.server1.http.timeout=30000

If "JAVA_HOME not defined" error occurs in /var/log/opsgenie/icinga2opsgenie.log, define it in /etc/opsgenie/profile shell script.

Use Marid to see rich alerts populated with host or service current status information in Opsgenie for every notification created in Icinga.

1091 952

For more information refer to Marid Integration Server and Callbacks docs. Please do not hesitate to get in touch with any questions, issues, etc.

🚧

Icinga integration package does not support SSL v1.0. If your Icinga Server has SSL v1.0, we suggest you to upgrade your SSL server.

FAQ and Troubleshooting

If the integration is not working, please check if the problem is mentioned below, and follow our advice:

1. Icinga alerts are not getting created in Opsgenie:

Run the following test command from the shell. Check if the test alert is created in Opsgenie:

/usr/bin/icinga2opsgenie -entityType=host -t=PROBLEM -hs=DOWN -hn=test_host
  • If a "Trace/breakpoint trap" error occurs: It means the icinga2opsgenie plugin being used isn't compatible with your server distribution. Follow the "Source and Recompiling icinga2opsgenie" section below and rebuild your icinga2opsgenie.go according to your specific server environment.
  • If the alert is created in Opsgenie: It means the integration is installed correctly. The problem might be that Icinga is not notifying the Opsgenie contact for alerts. Check the Icinga alert notifications log.
  • If not: Check the logs at /var/log/opsgenie/icinga2opsgenie.log. Look for the following errors in the log file:
  • If the error "RestException[Could not authenticate.]" is in the logs, it means Opsgenie couldn't identify the API key. Check if the API key is set correctly, as explained in "Opsgenie Plugin Configuration in Icinga" above.
  • If the error "Could not execute this action with apiKey of [Icinga] integration" is in the logs, make sure that the correct Icinga integration package was doanloaded and used, not Icinga 2 or any other.
  • If unsure of the problem, set the plugin's log level to debug, try again and send the logs to us at [email protected] so we can problem solve on our end and attempt to find a solution for you.
  • If there is no /var/log/opsgenie/icinga2opsgenie.log file, or there are no logs in it, check the following:
  • First, make sure the icinga user has permission to write to /var/log/opsgenie directory. The installation package automatically does this for you. If an error occurs, execute: chown -R icinga:opsgenie /var/log/opsgenie
  • Check the Icinga server logs at /usr/local/icinga/var/icinga.log. See if there are error logs regarding icinga2opsgenie, and contact us with them.

Setting icinga2opsgenie plugin's log level to DEBUG:

Change the line icinga2opsgenie.logger=warning to icinga2opsgenie.logger=debug in /etc/opsgenie/conf/opsgenie-integration.conf file.

2. The Icinga alert is not acknowledged when the alert is acked in Opsgenie:

  • First, check the alert logs.
  • If the "Posted [Acknowledge] action to Icinga.." action is not in the log, it means Opsgenie didn't send the Acknowledge action to Icinga. Check the integration configuration, as it might not have matched the alert action.
  • If "Executed [Acknowledge] action via Marid with errors." is seen in the log, it means the icingaActionExecutor.groovy script in your Marid has encountered an error. Check the logs at /var/log/opsgenie/marid/script.log for error logs.
  • If the "Posted [Acknowledge] action to Icinga.." action is not in the log and no related log after that, it might mean Marid is having connection problems. Check the logs at /var/log/opsgenie/marid/Marid.log for error logs.
  • If the problem cannot be identified, set Marid's script log level to debug, try again and send the /var/log/opsgenie/marid/script.log file to us at [email protected].

Setting Marid's script log level to DEBUG:

Change the line log4j.logger.script=WARN, script to log4j.logger.script=DEBUG, script in /etc/opsgenie/marid/log.properties file. Then, restart Marid service.

3. Marid is causing memory leak, or using up too much RAM:

Change the line log4j.rootLogger=WARN, marid to log4j.rootLogger=DEBUG, marid in /etc/opsgenie/marid/log.properties file. Then, restart Marid service and send the /var/log/opsgenie/marid/Marid.log file to us at [email protected] so we can analyze further.

Source and Recompiling icinga2opsgenie

The source for the executable icinga2opsgenie is located under /usr/bin/ and icinga2opsgenie.go is located under /etc/opsgenie/ and is also available at GitHub Opsgenie Integration repository. To change the behavior of the executable, edit icinga2opsgenie.go and build it using:

go build icinga2opsgenie.go

For installing go, refer to https://golang.org/doc/install. Note that the executable in the plugin is built for linux/386 systems.