Icinga2 Integration

What does Opsgenie offer to Icinga2 users?

Opsgenie Icinga2 integration plugin utilizes full capabilities of Opsgenie and provides a two-way integration with Icinga2.

Functionality of the integration

• Icinga2 sends alerts to Opsgenie with detailed information. Opsgenie acts as a dispatcher for Icinga2 alerts, determines the right people to notify based on on-call schedules– notifies via email, phone calls, text messages (SMS) and iPhone & Android push notifications, and escalates alerts until the alert is acknowledged or closed.
• Opsgenie automatically connects to Icinga2, gets performance data from Graphite for the host/service and attaches it to the alert.
• Opsgenie posts alert updates to Icinga2, e.g. acknowledging the alert will automatically ack the alert in Icinga2, alert comments are reflected to Icinga2 etc.

Installation

The steps below describe how to integrate Opsgenie and Icinga2 using Opsgenie Icinga2 integration plugin. Note that you may need to slightly alter these instructions depending on your exact Linux distribution and your Icinga2 configuration.

Prerequisites

Packages provided support the following systems:

• Red Hat based linux distributions
• Debian based linux distributions

Download OpsGenie Icinga2 Plugin

For Red Hat Based Distributions

• Download Opsgenie Icinga2 (Linux RPM)
• Run the following command :

rpm -i opsgenie-icinga2-<your_version>.rpm

🚧

During upgrades, rpm package does not overwrite your existing configurations. It saves the new default configuration file as opsgenie-integration.conf.rpmnew. You can find more information about rpm upgrade config file handling from here.

🚧

If you're updating the integration from version 1.., you must update your opsgenie-integration.conf file:

• Remove icinga.alert_histogram_image_url, icinga.trends_image_url, icinga.command_url properties.
• Add icinga.api_url (Icinga2 API endpoint of your Icinga server) and icinga.graphite_url properties.

For Debian Based Distributions

• Download Opsgenie Icinga2 (Linux DEB)
• Run the following command :

dpkg -i opsgenie-icinga2-<your_version>.deb

Add Icinga2 integration in Opsgenie

1. Please create an Opsgenie account if you haven't done so already.
2. Go to Opsgenie's Icinga2 Integration page.

🚧

For Free and Essentials plans, you can only add the integrations from the Team Dashboards, please use the alternative instructions given below to add this integration.

3. Specify who is notified of Icinga2 alerts using the Teams field. Auto-complete suggestions are provided as you type.

📘

An alternative for Step 2) and Step 3) is to add the integration from the Team Dashboard of the team which will own the integration. To add an integration directly to a team, navigate to the Team Dashboard and open Integrations tab. Click Add Integration and select the integration that you would like to add.

4. Click Save Integration. An "API Key" is generated for the integration. This key is used by Icinga2 to authenticate with Opsgenie and specify the integration to process Icinga2 alerts.

Opsgenie Plugin Configuration in Icinga2

The plugin uses a golang-executable file (included in the plugin as icinga2opsgenie) to create, acknowledge, and close alerts in Opsgenie. Icinga2 should be configured to execute this file on events to create, acknowledge, and close alerts in Opsgenie.

Setting the apiKey is required. Other configuration parameters are set to defaults that work with most Icinga2 implementations but may need to be modified as well.

There are three ways to configure golang-executable file:

1. Configuring from conf file: Configure from /etc/opsgenie/conf/opsgenie-integration.conf file. Configuring from conf file overwrites the configurations made in the script.
2. Configuring by using Golang Flags: Configure by entering flags to the command definition in /etc/icinga2/conf.d/opsgenie.conf file. Use -apiKey flag for the apiKey and -is flag for the icinga_server name. If multiple Icinga2 servers are not in use, it is not necessary to define the Icinga2 server. Using flags overwrites all the other configuration methods mentioned above.

📘

To send additional custom arguments define them as key-value-pairs in the arguments dictionary. Since Icinga2 shuffles the arguments, please use the order property to make sure that the custom arguments are placed at the end of the list.
Example usage:
"-spd" = "$service.perfdata$"
"customargument1" = {
value = "$customargument1$"
order = 1
}
Parse custom arguments by adding {{_payload.customArgName}} to wherever is needed on the input fields.
For more information about using raw parameters please visit Dynamic Fields document.

3. Configuring from script: Configure apiKey and icinga_server from icinga2opsgenie.go script. If this option is used, build the script again and put the new executable to /usr/bin directory. Find information about the location of the icinga2opsgenie.go and how to build a go script in the Source section.

📘

Configure the golang-executable to use a proxy for sending HTTP requests by defining the environment variable HTTP_PROXY=http://host:port

Configure Opsgenie to Icinga Integration (Optional)

🚧

If you are using Opsgenie Edge Connector instead of Marid, you can find the integration specific script and its sample config from here. For more information about OEC, please refer OEC Integration documentation

The plugin uses Marid utility (included in the plugin) to enrich alerts when they get created and to update the state of the them in Icinga2 when they get updated in Opsgenie. For example, when an alert is created in Opsgenie, Marid gets the details (histogram, trends etc.) from Icinga2 and attaches them to the alert. When users acknowledge an alert from their mobile devices using the Opsgenie app, the alert gets acknowledged in Icinga2, and when users add comments to alerts in Opsgenie, comments get posted to Icinga2 as well. Marid subscribes to alert actions in Opsgenie and reflects these actions on Icinga2 using Icinga2 API.

• To start Marid, run the following command:

/etc/init.d/marid start

• To stop Marid, run the following command:

/etc/init.d/marid stop

Marid is a java application; therefore requires the Java Runtime version 1.6+ Both the Open JDK and Oracle JVMs can be used.

📘

In order to use this feature "Send Alert Actions To Icinga2" checkbox should be enabled in Opsgenie Icinga2 Integration.

📘

Ensure that JAVA_HOME environment variable is set. If it is not, set it by removing the comment at the begining of the following line in /etc/opsgenie/profile file and set JAVA_HOME to your JRE installation directory.
#JAVA_HOME=<path/to/JDK or JRE/install>

To execute actions in Icinga2, Marid gets the configuration parameters from /etc/opsgenie/conf/opsgenie-integration.conf file.

Multiple Icinga2 Server Support

To configure Marid to forward Opsgenie alert actions to multiple Icinga2 servers, in order to do so :

• On each Icinga2 Server, modify /etc/opsgenie/conf/opsgenie-integration.conf icinga_server config property value to a unique name.
• On Marid server, modify /etc/opsgenie/conf/opsgenie-integration.conf, Add API endpoint, Graphite endpoint, user, password configuration for each Icinga2 server.
• opsgenie-integration.conf has a commented sample configuration for multiple Icinga servers :

#icinga.server1.api_url=http://icingaHost:port
#icinga.server1.graphite_url=http://icingaHost:port
#icinga.server1.user=icingaadmin
#icinga.server1.password=icingaadmin
#icinga.server1.http.timeout=30000

If a "JAVA_HOME not defined" error occurs in /var/log/opsgenie/icinga2opsgenie.log, define it in /etc/opsgenie/profile shell script.

If using Marid, rich alerts are seen populated with host or service current status information in Opsgenie for every notification created in Icinga2.

For more information refer to Marid Integration Server and Callbacks docs. Please do not hesitate to get in touch with any questions, issues, etc.

🚧

Icinga2 integration package does not support SSL v1.0. If your Icinga2 Server has SSL v1.0, we suggest you to upgrade your SSL server.

FAQ and Troubleshooting

If you're having trouble getting the integration to work, please check if your problem is mentioned below, and follow our advice:

1. Icinga 2 alerts are not getting created in Opsgenie:

Run the following test command from the shell. Check if the test alert is created in Opsgenie:

/usr/bin/icinga2opsgenie -entityType=host -t=PROBLEM -hs=DOWN -hn=test_host

• If a "Trace/breakpoint trap" error occurs: It means the icinga2opsgenie plugin isn't compatible with the server distribution. Follow the "Source and Recompiling icinga2opsgenie" section below and rebuild icinga2opsgenie.go according to the specific server environment.
• If the alert is created in Opsgenie: It means the integration is installed correctly. The problem might be that Icinga2 is not notifying the Opsgenie contact for alerts. Check the Icinga2 alert notifications log.
• If not: Check the logs at /var/log/opsgenie/icinga2opsgenie.log. Look for the following errors in the log file:
• If a "RestException[Could not authenticate.]" error is seen in the logs, it means Opsgenie couldn't identify the api key. Check if the API key correctly is set correctly, as explained in "Opsgenie Plugin Configuration in Icinga" above.
• If a "Could not execute this action with apiKey of [Icinga2] integration" error is seen in the logs, the wrong integration package may have been downloaded. Make sure the Icinga2 integration package is downloaded, not Icinga or any other.
• If the problem cannot be identified, set the plugin's log level to debug, try again and send the logs to us at [email protected].
• If there is no /var/log/opsgenie/icinga2opsgenie.log file, or there are no logs in it, check the following:
• First, make sure the Icinga user has permission to write to /var/log/opsgenie directory. The installation package automatically does this. If there is a problem, execute: chown -R icinga:opsgenie /var/log/opsgenie
• Now check the Icinga2 server logs at /var/log/icinga2/icinga2.log. See if there are error logs regarding icinga2opsgenie, and contact us with them.

Setting icinga2opsgenie plugin's log level to DEBUG:

Change the line icinga2opsgenie.logger=warning to icinga2opsgenie.logger=debug in /etc/opsgenie/conf/opsgenie-integration.conf file.

2. The Icinga2 alert is not acknowledged when you ack the alert at Opsgenie:

• First, check the alert logs.
• If the "Posted [Acknowledge] action to Icinga2.." action is not in the log, it means Opsgenie didn't send the Acknowledge action to Icinga2. Check the integration configuration, it might not have matched the alert action.
• If "Executed [Acknowledge] action via Marid with errors." error is seen in the log, it means the icingaActionExecutor.groovy script in your Marid has encountered an error. Check the logs at /var/log/opsgenie/marid/script.log for error logs.
• If only the "Posted [Acknowledge] action to Icinga2.." is seen in the log and no related log after that, it might mean Marid is having connection problems. Check the logs at /var/log/opsgenie/marid/Marid.log for error logs.
• If the problem cannot be identified, set the Marid's script log level to debug, try again and send the /var/log/opsgenie/marid/script.log file to us at [email protected].

Setting Marid's script log level to DEBUG:

Change the line log4j.logger.script=WARN, script to log4j.logger.script=DEBUG, script in /etc/opsgenie/marid/log.properties file. Then, restart Marid service.

3. Marid is causing memory leak, or using up too much RAM:

Change the line log4j.rootLogger=WARN, marid to log4j.rootLogger=DEBUG, marid in /etc/opsgenie/marid/log.properties file. Then, restart Marid service and send the /var/log/opsgenie/marid/Marid.log file to us at [email protected] so we can analyze further.

Source and Recompiling icinga2opsgenie

The source for the executable icinga2opsgenie is located under /usr/bin/ and icinga2opsgenie.go is located under /etc/opsgenie/ and is also available at GitHub Opsgenie Integration repository. To change the behavior of the executable, edit icinga2opsgenie.go and build it using:

go build icinga2opsgenie.go

For installing go, refer to https://golang.org/doc/install. Note that the executable in the plugin is built for linux/386 systems.