Zabbix Plugin

Opsgenie Zabbix integration plugin utilizes full capabilities of Opsgenie and provides bi-directional integration with Zabbix.

What does Opsgenie offer Zabbix users?

Integration leverages Opsgenie's Zabbix-specific executable and Marid utility to automatically create rich alerts and synchronize alert status between Zabbix and Opsgenie. Zabbix alarms are acknowledged automatically when the alert is acknowledged in Opsgenie. Opsgenie Zabbix plugin is the recommended integration method as it enables Zabbix to take advantage of advanced capabilities provided by Opsgenie.

Functionality of the integration

  • When users acknowledge an alert from their mobile devices using the Opsgenie app, the alert is acknowledged in Zabbix.

Installation

The steps below describe how to integrate Opsgenie and Zabbix using the Opsgenie Zabbix integration plugin. Note that these instructions may need slight alteration depending on the exact Linux distribution and the Zabbix configuration.

Prerequisites

Packages provided support the following systems:

  • Red Hat based linux distributions
  • Debian based linux distributions

Download Opsgenie Zabbix Plugin

For Red Hat Based Distributions

🚧

During upgrades, the rpm package does not overwrite existing configurations. It saves the new default configuration file as opsgenie-integration.conf.rpmnew. Find more information about rpm upgrade config file handling from here.

🚧

If you want to update from version 201X-XX-XX to 2.X.X, you must add --force parameter.

E.g.: rpm -U --force opsgenie-integration-<your_version>.rpm

We suggest that you backup your configuration files before update!

For Debian Based Distributions

Add Zabbix integration in Opsgenie

To add Zabbix integration in Opsgenie, go to Opsgenie Zabbix Integration page.

πŸ“˜

An alternative for this step is to add the integration from the Team Dashboard of the team which will own the integration. To add an integration directly to a team, navigate to the Team Dashboard and open Integrations tab. Click Add Integration and select the integration that you would like to add.

🚧

For Free and Essentials plans, you can only add the integrations from the Team Dashboards, please use the alternative instructions given above to add this integration.

Click on "Save Integration" button to save the integration. An "API Key" is generated for the integration. This key will be used by Zabbix to authenticate with Opsgenie and specify the integration that should be used to process Zabbix alerts.

2169

Opsgenie Plugin Configuration in Zabbix (for Zabbix version 3.0 and below)

The plugin uses a golang-executable file (included in the plugin as zabbix2opsgenie) to create, acknowledge, and close alerts in Opsgenie. Configure Zabbix to execute this file on events to create, acknowledge, and close alerts in Opsgenie.

Configure an action via the Zabbix frontend, that executes /etc/opsgenie/zabbix2opsgenie when an event is triggered.

Under Actions enable "Recovery Message" checkbox for closing the alert on Opsgenie when an alert is closed on Zabbix.

509

Under Operations, define a Remote Command that executes on Zabbix server. Paste the command in /etc/opsgenie/actionCommand.txt into Command.

Click Add then Save.

852

zabbix2opsgenie configuration parameters are found at /etc/opsgenie/conf/opsgenie-integration.conf file.

Configuration ParametersDescriptionMandatory to fill
apiKeyCopy the API key from the Zabbix integration created above. zabbix2opsgenie uses this key to authenticate to Opsgenie. API key is also used to identify the right integration configuration used to process alerts.Yes
teamsTeams field is used to specify which teams should be notified for the Zabbix alerts. This field is used to set the default teams field value. It can be modified to route different alerts to different teams in Opsgenie Zabbix integration, Advanced Settings page.Optional
recipientsRecipients field is used to specify who is notified of Zabbix alerts. This field is used to set the default recipients field value. The recipients can be modified to route different alerts to different people and teams in Opsgenie Zabbix integration, Advanced Settings page. Recipients can be set to users, groups, escalations or schedules who will be notified by Opsgenie. If you did not set recipients in the integration, this field is required.Optional
tagsTags field is used to specify the tags of the alert that created in Opsgenie.Optional
viaMaridUrlviaMaridUrl field is used to send alerts to Opsgenie through Marid. You should enter host and port values of your working Marid.

Useful when Zabbix server has no internet connection but Marid has internet connection.
In order to use this feature you should be running the Marid provided within Opsgenie Zabbix Plugin
Marid should be running with web server enabled ( http or https configurations eanbled )
Marid can run on a seperate host server, the communication between zabbix2opsgenie & Marid is done with basic http.
* Helps Zabbix server to consume less time when sending data to Opsgenie by letting Marid do the long task with an async approach.
Optional
logPathSpecifies the full path of the log file. (Default value is /var/log/opsgenie/zabbix2opsgenie.log)Optional
zabbix2opsgenie.http.proxy.enabledzabbix2opsgenie.http.proxy.enabled field is to enable/disable external proxy configuration. The default value is false.Optional
zabbix2opsgenie.http.proxy.hostIt is the host of the proxy.Optional
zabbix2opsgenie.http.proxy.portIt is the port of the proxy.Optional
zabbix2opsgenie.http.proxy.schemeIt is the proxy connection protocol. It may be http or https depending on your proxy servers. Its default value is http.Optional
zabbix2opsgenie.http.proxy.usernameIt is the Proxy authentication username.Optional
zabbix2opsgenie.http.proxy.passwordIt is the Proxy authentication password.Optional

Opsgenie Plugin Configuration in Zabbix (for Zabbix version 3.2 and above)

The plugin uses a golang-executable file (included in the plugin as zabbix2opsgenie) to create, acknowledge, and close alerts in Opsgenie. Configure Zabbix to execute this file on events to create, acknowledge, and close alerts in Opsgenie.

Configure an action via the Zabbix frontend, that executes/etc/opsgenie/zabbix2opsgenie when an event is triggered.

Select Configurations -> Actions then select Create New Action.

Select Operations under Actions.

1363

Under Operations, define a Remote Command that executes on Zabbix server. Paste the command in /etc/opsgenie/actionCommand.txt into Command.

1343

Select Recovery Operations under Actions.

Under Recovery Operations, define a Remote Command that executes on Zabbix server. Paste the command in /etc/opsgenie/actionCommand.txt into Command.

1351

Click Add and Save.

zabbix2opsgenie configuration parameters can be found at /etc/opsgenie/conf/opsgenie-integration.conf file.

Configuration ParametersDescriptionMandatory to fill
apiKey Copy the API key from the Zabbix integration you've created above. zabbix2opsgenie uses this key to authenticate to Opsgenie. API key is also used to identify the right integration configuration that should be used to process alerts.Yes
teams Teams field is used to specify which teams should be notified for the Zabbix alerts. This field is used to set the default teams field value. It can be modified to route different alerts to different teams in Opsgenie Zabbix integration, Advanced Settings page.Optional
recipients Recipients field is used to specify who should be notified for the Zabbix alerts. This field is used to set the default recipients field value. The recipients can be modified to route different alerts to different people and teams in Opsgenie Zabbix integration, Advanced Settings page. Recipients can be set to users, groups, escalations or schedules who will be notified by Opsgenie. If you did not set recipients in the integration, this field is required.Optional
tags Tags field is used to specify the tags of the alert that created in Opsgenie.Optional
viaMaridUrl viaMaridUrl field is used to send alerts to Opsgenie through Marid. You should enter host and port values of your working Marid.
Useful when Zabbix server has no internet connection but Marid has internet connection.
In order to use this feature you should be running the Marid provided within Opsgenie Zabbix Plugin
Marid should be running with web server enabled ( http or https configurations eanbled )
Marid can run on a seperate host server, the communication between zabbix2opsgenie & Marid is done with basic http.
* Helps Zabbix server to consume less time when sending data to Opsgenie by letting Marid do the long task with an async approach.
Optional
logPath Specifies the full path of the log file. (Default value is /var/log/opsgenie/zabbix2opsgenie.log)Optional
zabbix2opsgenie.http.proxy.enabled zabbix2opsgenie.http.proxy.enabled field is to enable/disable external proxy configuration. The default value is false.Optional
zabbix2opsgenie.http.proxy.host It is the host of the proxy.Optional
zabbix2opsgenie.http.proxy.port It is the port of the proxy.Optional
zabbix2opsgenie.http.proxy.scheme It is the proxy connection protocol. It may be http or https depending on your proxy servers. Its default value is http.Optional
zabbix2opsgenie.http.proxy.username It is the Proxy authentication username.Optional
zabbix2opsgenie.http.proxy.password It is the Proxy authentication password.Optional
opsgenie.api.urlIf you're using Opsgenie from another domain(eg. EU, sandbox), you should update this configuration.Optional

πŸ“˜

*If you want to send additional custom arguments, you can add them after the flags as: customArgName1 customArgValue1 customArgName2 customArgValue2

You can parse custom arguments by adding {{_payload.customArgName}} to wherever you need on the input fields.*

For more information about using raw parameters please visit the Dynamic Fields document.

Configure Opsgenie to Zabbix Integration (Optional)

🚧

If you are using Opsgenie Edge Connector instead of Marid, you can find the integration specific script and its sample config from here. For more information about OEC, please refer OEC Integration documentation

🚧

If you are using Zabbix version 4.0 or above, you need to configure the Marid config file by setting the alert action configuration as actions.Acknowledge.script=zabbixActionExecutorForZabbix4.groovy.

The plugin uses Marid utility (included in the plugin) to acknowledge alerts in Zabbix when alerts get acknowledged in Opsgenie. Marid subscribes to alert actions in Opsgenie and reflects these actions on Zabbix using Zabbix's JSON RPC api.

  • To start Marid, run the following command:/etc/init.d/marid start

  • To stop Marid, run the following command:/etc/init.d/marid stop

Marid is a java application; therefore requires the Java Runtime version 1.6+ Both the Open JDK and Oracle JVMs can be used.

πŸ“˜

Ensure that JAVA_HOME environment variable is set. If it is not, you may set it by removing the comment at the begining of the following line in /etc/opsgenie/profile file and set JAVA_HOME to your JRE installation directory.
#JAVA_HOME=<path/to/JDK or JRE/install>

To acknowledge alerts in Zabbix, Marid gets the configuration parameters from /etc/opsgenie/conf/opsgenie-integration.conf file.

Configuration Parameters
zabbix.command_url The Zabbix API endpoint to acknowledge Zabbix alerts.
zabbix.user The username that Marid used to authenticate to Zabbix.
zabbix.password The password that Marid uses to authenticate to Zabbix.

If you see "JAVA_HOME not defined" error in /var/log/opsgenie/zabbix2opsgenie.log, you should define it in /etc/opsgenie/profile shell script.

For more information refer to Marid Integration Server and Callbacks docs. Please do not hesitate to get in touch with any questions, issues, etc.

🚧

Zabbix integration package does not support SSL v1.0. If your Zabbix Server has SSL v1.0, we suggest you to upgrade your SSL server.

FAQ and Troubleshooting

If errors occur while getting the integration to work, please check if the problem is mentioned below, and follow our advice:

1. Zabbix alerts are not created in Opsgenie:

Run the following test command from the shell. Check if the test alert is created in Opsgenie:

/etc/opsgenie/zabbix2opsgenie -triggerName='zabbix test alert' -triggerStatus='PROBLEM'
  • If a "Trace/breakpoint trap" error occurs: It means the zabbix2opsgenie plugin isn't compatible with your server distribution. Follow the "Source and Recompiling zabbix2opsgenie" section below and rebuild the nagios2opsgenie.go according to a specific server environment.
  • If the alert is created in Opsgenie: It means the integration is installed correctly. The problem might be that Zabbix is not executing the command.
  • If not: Check the logs at /var/log/opsgenie/zabbix2opsgenie.log. Look for the following errors in the log file:
    • If "RestException[Could not authenticate.]" error occurs in the logs, it means Opsgenie couldn't identify the api key. Check if the API key is set correctly, as explained in "Opsgenie Plugin Configuration in Zabbix" above.
    • If unsure of the issue, set the plugin's log level to debug, try again and send the logs to us at [email protected].
  • If there is no /var/log/opsgenie/zabbix2opsgenie.log file, or there are no logs in it, check the following:
    • First, make sure the zabbix user has permission to write to /var/log/opsgenie directory. The installation package should automatically do this. If problems occur, execute: chown -R zabbix:opsgenie /var/log/opsgenie.
    • Now check the Zabbix server logs at /var/log/zabbix/zabbix_server.log. See if there are error logs regarding zabbix2opsgenie, and contact us with them.

Setting zabbix2opsgenie plugin's log level to DEBUG:

Change the line zabbix2opsgenie.logger=warning to zabbix2opsgenie.logger=debug in /etc/opsgenie/conf/opsgenie-integration.conf file.

2. The Zabbix alarm is not acknowledged when you ack the alert at Opsgenie:

  1. Check the Alerts logs.
  • If the "Posted [Acknowledge] action to Zabbix.." log does not appear, it means Opsgenie didn't send the Acknowledge action to Zabbix. Check the integration configuration, it might not have matched the alert action.
  • If "Executed [Acknowledge] action via Marid with errors." log occurs, it means the zabbixActionExecutor.groovy script in Marid has encountered an error. Check the logs at /var/log/opsgenie/marid/script.log for error logs.
  • If "Posted [Acknowledge] action to Zabbix.." log only occurs, and no related log after that, it might mean Marid is having connection problems. Check the logs at /var/log/opsgenie/marid/Marid.log for error logs.
  1. If unsure of the problem, set the Marid's script log level to debug, try again and send the /var/log/opsgenie/marid/script.log file to us at [email protected].

Setting Marid's script log level to DEBUG:

Change the line log4j.logger.script=WARN, script to log4j.logger.script=DEBUG, script in /etc/opsgenie/marid/log.properties file. Then, restart Marid service.

3. Marid is causing a memory leak, or using up too much RAM:

Change the line log4j.rootLogger=WARN, marid to log4j.rootLogger=DEBUG, marid in /etc/opsgenie/marid/log.properties file. Then, restart Marid service and send the /var/log/opsgenie/marid/Marid.log file to us at [email protected] so we can analyze further.

Source and Recompiling zabbix2opsgenie

The source for the executable zabbix2opsgenie, zabbix2opsgenie.go, is located under /etc/opsgenie/ and is also available at GitHub Opsgenie Integration repository. To change the behavior of the executable, edit zabbix2opsgenie.go and build it using:

go build zabbix2opsgenie.go

For installing go, refer to http://golang.org/doc/install. Note that the executable in the plugin is built for linux/386 systems.