Splunk Integration

Splunk makes it simple to collect, analyze and act upon the untapped value of the big data generated by your technology infrastructure, security systems, and business applications. Splunk has an alerting functionality.

Splunk alerts can be used to monitor for and respond to specific events. Alerts use a saved search to look for events in real time or on a schedule. Alerts trigger when search results meet specific conditions. Alert actions can be used to respond when alerts trigger.

What does Opsgenie offers Splunk users?

Opsgenie provides a two-way integration with Splunk. Splunk's Searching and Reporting app lets users search their data, create data models and pivots, save searches and pivots as reports, configure alerts, and create dashboards. Through Opsgenie's Alerts app, forward Splunk alerts to Opsgenie. With the Splunk Integration, Opsgenie acts as a dispatcher for these alerts, determines the right people to notify based on on-call schedules– notifies via email, text messages (SMS), phone calls, iOS & Android push notifications, and escalates alerts until the alert is acknowledged or closed. Through OEC (Opsgenie Edge Connector) you can forward Opsgenie alerts to Splunk as events for additional indexing and search.

Functionality of the integration

  • Opsgenie has a Splunk-specific alert app to send Splunk alerts to Opsgenie. Opsgenie also has a specific API for Splunk Integration, Splunk sends alerts through Opsgenie Alerts app to Opsgenie and Opsgenie handles the automatic creation of alerts.
  • If Create Splunk Events for Opsgenie Alerts is enabled, alert specific actions (Create Alert, Acknowledge Alert, etc.) will be sent to Splunk as events through OEC.

Add Splunk Integration in Opsgenie

  1. Please create an Opsgenie account if you haven't done so already.
  2. Go to Opsgenie's Splunk Integration page.

🚧

For Free and Essentials plans, you can only add the integrations from the Team Dashboards, please use the alternative instructions given below to add this integration.

  1. Specify who is notified for Splunk alerts using the Teams field. Auto-complete suggestions are provided as you type.

📘

An alternative for Step 2) and Step 3) is to add the integration from the Team Dashboard of the team which will own the integration. To add an integration directly to a team, navigate to the Team Dashboard and open Integrations tab. Click Add Integration and select the integration that you would like to add.

  1. Copy the API Key.
  2. Click Save Integration.

Configuration in Splunk

  1. In Splunk, install the Opsgenie App from Splunkbase.
  1. After installation, navigate to "Apps" and click Set Up to configure the Opsgenie App.
  1. Paste the API Key copied previously in the chapter:Add Splunk Integration in Opsgenie

🚧

Updating API Key

If using Splunk Cloud and need to update the API Key you've set in this step, please file a Splunk support case to uninstall the app.

  1. Run a search in Splunk to create an Alert.
  2. Click Save As and select "Alert" from the dropdown list.
  1. Populate alert title and specify conditions.
  2. Click the +Add Actions button to access the dropdown list and select "Opsgenie".
  1. Click Save.

Integration via OEC

Use Opsgenie's OEC and the Splunk script to update alerts on Splunk. This enables deployment of your own scripts/ability to modify the ones provided and execute customized actions on Splunk.

To use the Splunk integration package, follow the steps in the sections below through the end.

👍

Download latest version

For downloading latest version of Splunk packages, please use readme file of oec-scripts repository

Installation

For Red Hat Based Distributions

  • Run following command : rpm -i opsgenie-splunk-<your_version>.rpm

For Debian Based Distributions

  • Run following command : dpkg -i opsgenie-splunk-<your_version>.deb

For Windows

  • Unzip opsgenie integration zip file which contains OEC package into a directory (C:\opsgenie\oec is the preferred path.) Follow the instructions here to install OEC on Windows.

  • In order to learn more about how to run OEC you can refer to the Running OEC documentation.

To be able to execute actions in Splunk, OEC gets the configuration parameters from the configuration file. The configuration file can be found under /home/opsgenie/oec/conf/config.json and for windows C:\opsgenie\oec\conf\config.json

Configuring Splunk Integration for OEC

  • In order to use OEC utility for your Splunk integration, Send Via OEC in your integration settings.
  • Configurations regarding Splunk are optional if you select send via OEC, you can choose to configure them in your OEC config file.

Configuring Splunk Integration for Splunk Cloud

  • If you are using Splunk Cloud, Opsgenie can now deliver events to your Splunk Cloud instance.
  • DO NOT select Send Via OEC and enter your Splunk Cloud Url and Splunk Token in the respective fields. Please enter Splunk Cloud Url along with port.

Configuration in Splunk

  1. Log in to Splunk as an administrator.
  2. From the home page, select Add Data.
  1. From bottom, select Monitor.
  1. From data input options, select HTTP Event Collector and give your new event collector a name.
  1. Complete rest of the steps without modification unless desired. In final step note the token value given by Splunk.
  1. To enable data input through HTTP, select Settings > Data Input from top bar, then navigate to HTTP Event Collector. Afterwards make sure All Tokens is set to enabled in Global Settings menu.

Now that collector is configured, Splunk will create events whenever Opsgenie issues alert related actions.

OEC Configuration

To be able to create events in Splunk, OEC gets the configuration parameters from integration settings.

Configuration Parameter

Description

Location

Splunk URL

URL of your Splunk HTTP Event Collector Server including port. For ex: http://<splunk_server>:

/home/opsgenie/oec/conf/config.json

Splunk Token

Token of your Splunk HTTP Event Collector data input

/home/opsgenie/oec/conf/config.json

Verify SSL

The request to your Splunk Server might fail if the SSL certificate verification fails. You can choose to not to verify ssl cert, default is false

/home/opsgenie/oec/conf/config.json

Sample payload sent from Splunk and draggable fields in Opsgenie

The result field of the content below differs according to the fields of the lines that match the search. That's why Opsgenie provides common fields of the result object in available fields. Raw, Index, Serial, Source Type etc. are examples of the common fields.

We also added "Result Object" to the available fields, to make it possible to extract custom fields from the result object.

For example, to put the date_month field of result object to the alert: {{result.date_month}} solves the problem.

Create Alert payload:

{
    "session_key": "r41vK7psTN9iIp1HQXqgNxTHPz2AW_Ee3ELbdYM4FBqiBbI7L6f82o6f6IENt6Q_Xdq2V4jBSkjkyIfXIm56xbbcFcpWlcJNB0ZUZaezsImsTQ2lGWH26yiZ8l854Or8SPETrWuVgTKVeC",
    "search_name": "fail",
    "results_link": "http://Tuba-MacBook-Pro.local:8000/app/search/search?q=%7Cloadjob%20rt_scheduler__admin__search__fail_at_1464802733_32.0%20%7C%20head%201%20%7C%20tail%201&earliest=0&latest=now",
    "app": "search",
    "sid": "rt_scheduler__admin__search__fail_at_1464802733_32.0",
    "configuration": {
        "api_url": "http://4kmm916oxm9m.runscope.net"
    },
    "server_host": "Tuba-MacBook-Pro.local",
    "owner": "admin",
    "results_file": "/Applications/Splunk/var/run/splunk/dispatch/rt_scheduler__admin__search__fail_at_1464802733_32.0/per_result_alert/tmp_0.csv.gz",
    "server_uri": "https://127.0.0.1:8089",
    "result": {
        "date_month": "may",
        "index": "main",
        "_indextime": "1464802756",
        "date_minute": "15",
        "date_hour": "0",
        "splunk_server": "Tuba-MacBook-Pro.local",
        "date_mday": "11",
        "sourcetype": "secure",
        "source": "tutorialdata copy 2.zip:./www1/secure.log",
        "date_second": "2",
        "_serial": "0",
        "_sourcetype": "secure",
        "date_year": "2016",
        "eventtype": "",
        "_kv": "1",
        "timeendpos": "25",
        "timestartpos": "4",
        "linecount": "1",
        "date_zone": "local",
        "date_wday": "wednesday",
        "punct": "____::__[]:________...___",
        "_raw": "Thu May 11 2016 00:15:02 www1 sshd[4747]: Failed password for invalid user jabber from 118.142.68.222 port 3187 ssh2",
        "_eventtype_color": "",
        "_confstr": "source::tutorialdata copy 2.zip:./www1/secure.log|host::Tuba-MacBook-Pro.local|secure",
        "_time": "1462914902",
        "host": "Tuba-MacBook-Pro.local"
    }
}

Sample payload sent to Splunk

{ 
    "alertDetails":{ 

    },
    "customerDomain":"opsgenie",
    "sendViaMarid":false,
    "alertId":"2ce35c31-a8e1-4be5-a781-110564a45e75-1568102450408",
    "url":"http://localhost:8088",
    "type":"Splunk",
    "alert":{ 
        "alertId":"2ce35c31-a8e1-4be5-a781-110564a45e75-1568102450408"
    },
    "source":null,
    "token":"4c40855d-a361-4a9b-984f-0ab46c91a35f",
    "params":{ 
        "alertDetails":{ 

        },
        "customerDomain":"opsgenie",
        "sendViaMarid":false,
        "alertId":"2ce35c31-a8e1-4be5-a781-110564a45e75-1568102450408",
        "source":null,
        "url":"http://localhost:8088",
        "integrationName":"Splunk",
        "alert":{ 
            "alertId":"2ce35c31-a8e1-4be5-a781-110564a45e75-1568102450408"
        },
        "integrationId":"c345dce0-50e3-4498-9433-1fddd3f4f1fd",
        "token":"4c40855d-a361-4a9b-984f-0ab46c91a35f",
        "integrationType":"Splunk",
        "action":"Create",
        "mappedActionV2":{ 
            "extraField":"",
            "name":"createEvent"
        },
        "type":"Splunk",
        "customerId":"5035e5cd-4791-4995-9154-037027f8e0b6"
    },
    "action":"Create",
    "mappedActionV2":{ 
        "extraField":"",
        "name":"createEvent"
    },
    "integrationId":"c345dce0-50e3-4498-9433-1fddd3f4f1fd",
    "integrationType":"Splunk",
    "integrationName":"Splunk",
    "customerId":"5035e5cd-4791-4995-9154-037027f8e0b6"
}

Sample Alert

Contact

Contact us for your questions and support requests.

Updated 23 days ago


Splunk Integration


Splunk makes it simple to collect, analyze and act upon the untapped value of the big data generated by your technology infrastructure, security systems, and business applications. Splunk has an alerting functionality.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.