What does Opsgenie offer OP5 Monitor users?

Opsgenie provides a powerful bidirectional integration with OP5. When a new alert - host or service - is created in OP5, a corresponding Opsgenie alert is automatically created, containing rich information about the alert. Opsgenie provides detailed notifications with on-call rotations, scheduling features, and alert escalations. Users choose any of the alert actions of Opsgenie to map to the acknowledge action of OP5.

Functionality of the integration

OP5 to Opsgenie:

• When a host or service alert is created on OP5, an alert is created in Opsgenie.
• When a host or service alert is closed on OP5, the corresponding alert is closed in Opsgenie.
• When a host or service alert is acknowledged on OP5, the corresponding alert is acknowledged in Opsgenie.

Opsgenie to OP5:

• If Send Alert Updates Back to OP5 is enabled, actions executed on OP5 alerts are sent to OP5 as acknowledgment action.
• This action mapping features are explained in detail in the Action Mapping Feature section below.

Download Opsgenie OP5 Package

For OP5 Monitor's Red Hat Based Distributions

• Download Opsgenie OP5 (Linux RPM)
• Either share the OP5 file to the remote OP5 monitor or copy the link in the opening page and run the following command:
  wget https://linkTo.the.OP5Package
• Run the following command :
  rpm -i opsgenie-op5-<your_version>.rpm

🚧

During upgrades, the rpm package does not overwrite existing configurations. It saves the new default configuration file as opsgenie-integration.conf.rpmnew. Find more information about rpm upgrade config file handling from here.

🚧

To update from version 201X-XX-XX to 2.X.X, you must add --force parameter. E.g.:
rpm -U --force opsgenie-integration-<your_version>.rpm

We suggest that you backup your configuration files before updating!

Adding OP5 Integration in Opsgenie

1. Please create an Opsgenie account if you haven't done so already.
2. Go to OP5 Integration page.

🚧

For Free and Essentials plans, you can only add the integrations from the Team Dashboards, please use the alternative instructions given below to add this integration.

3. Specify who is notified of OP5 alerts using the Teams field. Auto-complete suggestions are provided as you type.

📘

An alternative for Step 2) and Step 3) is to add the integration from the Team Dashboard of the team which will own the integration. To add an integration directly to a team, navigate to the Team Dashboard and open Integrations tab. Click Add Integration and select the integration that you would like to add.

4. Copy the integration API key to use for Marid configuration.
5. Click Save Integration.

Opsgenie OP5 Package Configuration in OP5 Monitor

The plugin uses a golang-executable file (included in the plugin as nagios2opsgenie) to create, acknowledge, and close alerts in Opsgenie. Configure OP5 to execute this file on events to create, acknowledge, and close alerts in Opsgenie.

Setting the apiKey is required. Other configuration parameters are set to defaults that work with OP5.

There are three ways to configure golang-executable files:

1. Configuring from conf file: Configure from /etc/opsgenie/conf/opsgenie-integration.conf file. Configuring from conf file will overwrite the configurations made in the script.
2. Configuring by using Golang Flags: Configure by entering flags to command in the opsgenie.cfg file. Use -apiKey flag for the apiKey and -ns flag for nagios_server name. If multiple OP5 servers are not in use, there is no need to define the nagios server. Using flags overwrites all the other configuration methods mentioned above.

Configure the apiKey from the cfg file as follows:

define command {
    command_name    notify-service-by-opsgenie
    command_line    /usr/bin/nagios2opsgenie -apiKey="apiKey1" -entityType=service ...
}

When apiKey is added to the cfg file, it overrides the apiKey in the opsgenie-integration.conf file.

📘

To send additional custom arguments, add them after the flags as: customArgName1 customArgValue1 customArgName2 customArgValue2
Parse custom arguments by adding {{_payload.customArgName}} to wherever is needed in the input fields.
For more information about using raw parameters please visit Dynamic Fields document.

3. Configuring from script: Configure apiKey and nagios_server from nagios2opsgenie.go script. To use this option, build the script again and put the new executable to /usr/bin directory. Find information about the location of the nagios2opsgenie.go and how to build a go script in the "Source" section.

Define Opsgenie as Contact

1. Login to your OP5 monitor.
2. Go to your /opt/monitor/etc directory where you'll find nagios.cfg.
   cd /opt/monitor/etc
3. Add the following line to main Nagios configuration file (nagios.cfg)
   

...
cfg_file=opsgenie.cfg
...

4. Reboot the monitor to see the "opsgenie" contact in the contact list.
5. Add the contact "opsgenie" to the OP5 Monitor’s main contact group.

If everything goes well, alerts are created in Opsgenie for every notification created in OP5.

Configure Opsgenie to OP5 Integration (Optional)

🚧

If you are using Opsgenie Edge Connector instead of Marid, you can find the integration specific script and its sample config from here. For more information about OEC, please refer OEC Integration documentation

The plugin uses Marid utility (included in the plugin) to enrich alerts when they get created and to update the state of the them in OP5 when they get updated in Opsgenie. For example, when users acknowledge an alert from their mobile devices using the Opsgenie app, alert gets acknowledged in OP5, and when users add notes to alerts in Opsgenie, comments get posted to OP5 as well. Marid subscribes to alert actions in Opsgenie and reflects these actions on OP5.

• To start Marid, run the following command:
  /etc/init.d/marid start
• To stop Marid, run the following command:
  /etc/init.d/marid stop

📘

There is also a restart command for Marid: /etc/init.d/marid restart

Marid is a Java application; therefore requires the Java Runtime version 1.6+ Both the Open JDK and Oracle JVMs can be used.

📘

In order to use this feature "Send Alert Actions To OP5" checkbox should be enabled in Opsgenie OP5 Integration.

📘

Ensure that JAVA_HOME environment variable is set. If it is not, you may set it by removing the comment at the begining of the following line in /etc/opsgenie/profile file and set JAVA_HOME to your JRE installation directory.

#JAVA_HOME=<path/to/JDK or JRE/install>

To execute actions in OP5, Marid gets the configuration parameters from /etc/opsgenie/conf/opsgenie-integration.conf file.

If using Marid, rich alerts are populated with host or service current status information in Opsgenie for every notification created in OP5.

For more information refer to Marid Integration Server and Callbacks docs. Please do not hesitate to get in touch with any questions, issues, etc.

FAQ and Troubleshooting

If the integration is not working, please check if the problem being experienced is mentioned below, and follow our advice:

1- OP5 alerts are not getting created in Opsgenie:

Run the following test command from the shell. Check if the test alert is created in Opsgenie:

/usr/bin/nagios2opsgenie -entityType=host -t=PROBLEM -hs=DOWN -hn=test_host

If a "Trace/breakpoint trap" error occurs: It means the nagios2opsgenie plugin isn't compatible with the server distribution. Follow the "Source and Recompiling nagios2opsgenie" section below and rebuild nagios2opsgenie.go according to the specific server environment.

If the alert is created in Opsgenie: It means the integration is installed correctly. The problem might be that OP5 is not notifying the Opsgenie contact for alerts. Check the OP5 alert notifications log.

If not: Check the logs at /var/log/opsgenie/nagios2opsgenie.log. Look for the following errors in the log file:

• If a "RestException[Could not authenticate.]" error appears in the logs, it means Opsgenie couldn't identify your API key. Check if the API key is set correctly, as explained in "Opsgenie OP5 Package Configuration in OP5 Monitor" above.
• If "Could not execute this action with apiKey of [OP5] integration" appears in the logs, the wrong integration package may have beend downloaded. Make sure to download the OP5 integration package.
• If unsure of the problem, set the plugin's log level to debug, try again and send the logs to us at [email protected]

If there is no /var/log/opsgenie/nagios2opsgenie.log file, or there are no logs in it, check the following:

1. First, make sure the nagios user has permission to write to /var/log/opsgenie directory. The installation package automatically does this for you. If you encounter problems, execute chown -R nagios:opsgenie /var/log/opsgenie.
2. Now check the OP5 server logs at /var/log/opsgenie/nagios2opsgenie.log. See if there are error logs regarding nagios2opsgenie, and contact us with them.

Setting nagios2opsgenie plugin's log level to DEBUG:

Change the line nagios2opsgenie.logger=warning to nagios2opsgenie.logger=debug in /etc/opsgenie/conf/opsgenie-integration.conf file.

2- The OP5 alert is not acknowledged when acked in Opsgenie:

1. First, check your alert logs.

• If a "Posted [Acknowledge] action to OP5.." error does not appear in the log, it means Opsgenie didn't send the Acknowledge action to OP5. Check the integration configuration, it might not have matched the alert action.
• If a "Executed [Acknowledge] action via Marid with errors." error appears in the log, it means the op5ActionExecutor.groovy script in Marid has encountered an error. Check the logs at /var/log/opsgenie/marid/script.log for error logs.
• If the "Posted [Acknowledge] action to OP5.." is the only error to appear in the log and no related log after that, it might mean Marid is having connection problems. Check the logs at /var/log/opsgenie/marid/Marid.log for error logs.

🚧

If no logs exist, restart Marid and try sending an Acknowledge action again.

2. If unsure of the problem, set the Marid's script log level to debug, try again and send the /var/log/opsgenie/marid/script.log file to us at [email protected]

Setting Marid's script log level to DEBUG:

Change the line log4j.logger.script=WARN, script to log4j.logger.script=DEBUG, script in /etc/opsgenie/marid/log.properties file. Then, restart Marid service.

3- Marid is causing memory leak, or using up too much RAM:

Change the line log4j.rootLogger=WARN, marid to log4j.rootLogger=DEBUG, marid in /etc/opsgenie/marid/log.properties file. Then, restart Marid service and send the /var/log/opsgenie/marid/Marid.log file to us at [email protected] so we can analyze further.

4- If "no java executable is found" error occurs while installing Opsgenie OP5 Package :

To get Marid to work Java is needed. If it's not present on the OP5 monitor install it via:
yum install java

Then run the following command to find the java:
ls -l /usr/lib/jvm

Set the JAVA_HOME to the directory containing a bin/java executable using export command:
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.e16_8.x86_64

Source and Recompiling nagios2opsgenie

The source for the executable nagios2opsgenie is located under /usr/bin/ and nagios2opsgenie.go is located under /etc/opsgenie/ and is also available at GitHub Opsgenie Integration repository. To change the behavior of the executable, edit nagios2opsgenie.go and build it using:
go build nagios2opsgenie.go

Refer to http://golang.org/doc/install/source. Note that nagios2opsgenie is built for linux/amd64 systems.

Action Mapping Feature

Opsgenie allows customers to define action mappings between Opsgenie actions and OP5 actions.

To use the Action Mappings feature, provide the information listed below to the integration:

• Username: The username of your OP5 account.
• Password: The password of your OP5 account.
• OP5 Monitor URL: This is the domain name of your OP5 Monitor e.g. https://19.167.1.143

For alerts created by OP5

• Use this section to map Opsgenie actions to OP5 actions when the source of the alert is OP5 (i.e When the alert is created by OP5 integration.)
• Map different Opsgenie actions to different OP5 actions. For example, acknowledge the alert in OP5, when the alert is acknowledged from OP5. In order to do this, define If alert is acknowledged in Opsgenie, acknowledge in OP5. mapping in Send Alert Updates Back to OP5 section.