Datadog Integration

Datadog is a popular cloud and application monitoring service. With Datadog, gain insight into granular metrics from entire stacks, from application-level to actual hosts and VMs.

What does OpsGenie offer Datadog users?

OpsGenie has a native, powerful bidirectional integration with Datadog. Use the integration to automatically sync your Datadog alerts with OpsGenie alerts, and benefit from OpsGenie's rich alert notification system, escalations, and on-call rotations.

Functionality of the integration

  • Datadog triggers an alert when a defined condition is matched. When an alert is created in Datadog, an alert is also created in OpsGenie automatically through the integration.
  • Create alerts in OpsGenie from Datadog UI by @mentioning OpsGenie service in the text.
  • When the alert is acknowledged in OpsGenie, alert is acknowledged automatically in Datadog as well.
  • When the alert is closed in OpsGenie, alert is closed automatically in Datadog as well.
  • When an alert is created in OpsGenie from any source (other than Datadog), OpsGenie is able to post the alert to Datadog (optional).
  • Acknowledge (or close) the alerts from Datadog using Datadog's @opsgenie-acknowledge and @opsgenie-close commands in a comment.

Add Datadog Integration to OpsGenie

  1. Please create an OpsGenie account if you haven't done so already
  2. Go to OpsGenie Datadog Integration page.

For Essentials plan, you can only add the integrations from the Team Dashboards, please use the alternative instructions given below to add this integration.

  1. Specify who is notified for Datadog alerts using the Teams field. Auto-complete suggestions are provided as you type.

An alternative for Step 2) and Step 3) is to add the integration from the Team Dashboard of the team which will own the integration. To add an integration directly to a team, navigate to the Team Dashboard and open Integrations tab. Click Add Integration and select the integration that you would like to add.

  1. Paste the Datadog API key obtained while configuring Datadog.
  2. Click Save Integration.

Configuration in Datadog

  1. In Datadog, select Integrations -> Integrations from the left pane.
  2. Choose OpsGenie from the Available list.
  3. Select Configurations tab.
  4. Enter the name of the integration and paste the API key that you copied from OpsGenie.
  5. Click Install Integration.

To test the integration, simply make a post with @opsgenie (i.e. @opsgenie This is a test alert!) in the "Events" page. This creates an alert in OpsGenie and shows up in your Datadog Events page.

Sample payloads sent from Datadog to OpsGenie

Metric alert payload:

    {
        "event_url": "",
        "last_updated": 1397628633000,
        "event_type": "metric_alert",
        "snapshot_link": "https://app.datadoghq.com/alert/alert_state?alert_id=51950&to_ts=1397628600000&aggr=avg&scope=host%3Aismet-if&metric=system.disk.free&from_ts=1397627940000",
        "msg_title": "[Triggered] system.disk.free",
        "monitor_data": {
            "snap_url": "https://p.datadoghq.com/snapshot/view/dd-snapshots-prod/org_2305/2014-07-23/6803a21774c63347c032d92a2aff01ababfba231.png",
            "group": "guruop",
            "name": "aws.dynamodb.returned_item_count",
            "type": "metric alert",
            "group_key": "key5",
            "query":"min(last_5m):avg:system.cpu.idle{*} < 99",
            "message": "@opsgenie-service_1",
            "org_id": 2305,
            "id": 59482,
            "metric": "system.cpu.idle"
        },
        "dd_tags": [
            "alert",
            "tag2"
        ],
        "message": "%%%\n@opsgenie-team \n\n[![Metric Graph](https://p.datadoghq.com/snapshot/view/dd-snapshots-prod/org_8283/2014-04-16/7103f56dfa041445fb099f014ca5a3fce0cfcaea.png)](https://app.datadoghq.com/alert/alert_state?alert_id=51950&to_ts=1397628600000&aggr=avg&scope=host%3Aismet-if&metric=system.disk.free&from_ts=1397627940000)\n\n**system.disk.free** over **host:ismet-if** was **> 1** on average during the **last 10m**.\n\n\n%%%",
        "event_id": "2237506288020066106",
        "snapshot_url": "https://p.datadoghq.com/snapshot/view/dd-snapshots-prod/org_8283/2014-04-16/7103f56dfa041445fb099f014ca5a3fce0cfcaea.png",
        "priority": "normal",
        "alias": "ad07c3317c7a1f9a35ea4ac2fdb5d706",
        "msg_text": "%%%\n@opsgenie-team \n\n[![Metric Graph](https://p.datadoghq.com/snapshot/view/dd-snapshots-prod/org_8283/2014-04-16/7103f56dfa041445fb099f014ca5a3fce0cfcaea.png)](https://app.datadoghq.com/alert/alert_state?alert_id=51950&to_ts=1397628600000&aggr=avg&scope=host%3Aismet-if&metric=system.disk.free&from_ts=1397627940000)\n\n**system.disk.free** over **host:ismet-if** was **> 1** on average during the **last 10m**.\n\n\n%%%",
        "action": "create",
        "templated_message": "temp msg",
        "source_type_name": "Monitor Alert"
    }

This payload is parsed by OpsGenie as:

[
          "last_updated": 1397628633000,
          "message_title": "[Triggered] system.disk.free",
          "query": "min(last_5m):avg:system.cpu.idle{*} < 99",
          "snapshot_url": "https://p.datadoghq.com/snapshot/view/dd-snapshots-prod/org_2305/2014-07-23/6803a21774c63347c032d92a2aff01ababfba231.png",
          "message": "@opsgenie-service_1",
          "priority": "normal",
          "event_url": "",
          "monitor_name": "aws.dynamodb.returned_item_count",
          "snapshot_link": "https://app.datadoghq.com/alert/alert_state?alert_id=51950&to_ts=1397628600000&aggr=avg&scope=host%3Aismet-if&metric=system.disk.free&from_ts=1397627940000",
          "templated_message": "temp msg",
          "event_type": "metric_alert",
          "event_id": "2237506288020066106",
          "metric": "system.cpu.idle",
          "alias": "ad07c3317c7a1f9a35ea4ac2fdb5d706",
          "action": "create",
          "dd_tags": [
            "alert",
            "tag2"
          ],
          "group": "guruop",
          "group_key": "key5",
          "source_type_name": "Monitor Alert"
    ]

@opsgenie-acknowledge command payload:

   {
      "last_updated": "1456994681000",
      "msg_title": "null",
      "user_name": "halit okumus",
      "user_handle": "halit.okumus@ifountain.com",
      "snapshot_url": "null",
      "source": "null",
      "msg_text": "@opsgenie-acknowledge I am on it",
      "priority": "normal",
      "message": "@opsgenie-acknowledge I am on it",
      "event_url": "",
      "snapshot_link": "null",
      "dd_tags": [
            "alert",
            "tag2"
        ],
      "event_type": "api",
      "event_id": "3233503302758961154",
      "monitor_data": "null",
      "alias": "f769ea8a-1e8c-48c1-943f-b51792e14de8",
      "action": "acknowledge",
      "alertId": "f769ea8a-1e8c-48c1-943f-b51792e14de8"
    }

@opsgenie-close command payload:

 {
      "last_updated": "1456998760000",
      "msg_title": "null",
      "user_name": "halit okumus",
      "user_handle": "halit.okumus@ifountain.com",
      "snapshot_url": "null",
      "source": "null",
      "msg_text": "@opsgenie-close closing this one",
      "priority": "normal",
      "message": "@opsgenie-close closing this one",
      "event_url": "",
      "snapshot_link": "null",
      "dd_tags": [
            "alert",
            "tag2"
        ],
      "event_type": "api",
      "event_id": "3233571736653926401",
      "monitor_data": "null",
      "alias": "f769ea8a-1e8c-48c1-943f-b51792e14de8",
      "action": "close",
      "alertId": "f769ea8a-1e8c-48c1-943f-b51792e14de8"
    }

Sample payload sent from OpsGenie to Datadog

{
  "apiKey": "2980d945-18b7-4371-9299-7ff60c21082c",
  "alert": {
    "createdAt": 1456756398904,
    "tinyId": "48",
    "alias": "alertAlias",
    "alertId": "5c1173aa-bbdb-4e32-b393-968b803b43d3",
    "source": "halit@opsgenie.com",
    "message": "OpsGenie test alert",
    "entity": "alertEntity",
    "updatedAt": 1456756398904000500,
    "username": "halit@opsgenie.com",
    "insertedAt": 1456756398904000500,
    "description": "test alert description",
    "tag": [
      "tag1",
      "tag2"
    ],
    "note": "test comment"      //only on AddNote action, note is put here as an example
  },
  "integrationName": "Datadog",
  "action": "Create",       //action can be Create, Close, AddNote or Acknowledge
  "integrationId": "2da74f28-3288-49e9-bad5-c97775e8b247",
  "source": {
    "name": "web",
    "type": "API"
  }
}

Sample alert