Rate Limiting

To protect the system from being overwhelmed, to provide more stable throughput and to provide fairness among different accounts in terms of resource consuming, Opsgenie applies rate limiting to the API and integration requests. If a request exceeds the defined rate limit, the request is either rejected with the HTTP status 429 - Too Many Requests or delayed to be executed.

Rate Limiting Strategy

Token Bucket Algorithm is used for rate limiting, where each received request corresponds to a single token. The limit of the request rate, the size of the bucket and the rate of adding a new token, is defined according to the following factors:

Opsgenie splits received requests into the following categories in terms of request domains:

• Alert: It's the domain for any request to Alerts or Alert API v1 endpoints.
• Heartbeat: It's the domain for any request to Heartbeat Rest API or Heartbeat API v1 endpoints.
• Integration: It's the domain for any request except the ones to Rest API and API v1 endpoints (Web API deprecated).
• Configuration: It's the domain for any request to the remaining Rest API or API v1 endpoints (Web API deprecated) like Team API, User API, Integration API, Escalation API, etc.
• Incident: It's the domain for any request to Incident API endpoints.
• Email: It's the domain for any email received via Email Integrations or Heartbeats.
• Search: It's the domain for any API request made to the list alerts and users.
• Schedule-oncalls: It's the domain for any request made to retrieve the current and next on-call user(s).
• Integration Configuration: It's the domain for create/update/delete integration requests except read and list requests.

📘

Please note: that the rate limiting is applied separately for each request domain. In other words, exceeding the rate limit for a domain doesn't mean that the rate limit is exceeded for other domain(s).

Opsgenie checks the rate limit status of any request to any domain in terms of two different time interval units:

• Minutes
• Seconds

When a request is received, Opsgenie decides the request status by checking the allowed number of requests per minute and the allowed number of requests per second separately. Please note: that the time interval is tracked as a rolling window.

Time unit interval types (that are minutes and seconds) form a separate bucket for each request domain according to the Token Bucket Algorithm and there are two buckets for each domain as a matter of course. When a request for a domain exceeds the rate limit of at least one of the related buckets, the request is either throttled or delayed.

To summarize, when Opsgenie receives a request, the limits for both minute rate and second rate are checked first. If either of the limits are exceeded, the request is throttled (or delayed).

📘

The request rate limits are account-wide and the requests from any integration of your account share the same limits by default. However, Opsgenie can apply integration fairness that will prevent each integration from consuming a defined percentage of the account-wide rate limits. You can contact us to activate integration fairness for your account.

There are 3 possible rate-limiting states for each request:

Rate Limit Calculation

📘

BaseLimit is the pre-defined number of allowed requests to the related domain for the time interval. Base limit is defined according to the following factors:

• Time Interval Unit: The base limit of the number of requests per minute is higher than the base limit of the number of requests per second.
• Subscription Plan Type: Each subscription plan has a different minimum rate limit. Please note: that Enterprise plan has the highest minimum limit and free plans have the lowest minimum limit. You can refer here for further information about our subscription plans.
• Request Domain: The rate limits of the requests to Configuration domain are lower than the other 3 domains.

(#OfUsers is the number of purchased user seats in your account subscription.)

For example, assume that your account is subscribed to Standard plan with 60 users. If the base limit of requests per minute is 500 and the base limit of requests per second is 100, the rate limits for your account would be as follows:

• 740 requests per minute
• 101 requests per second

Additionally, if integration fairness is activated with 10% rate for your account (upon your request), the rate limit for each integration would be as follows:

• 74 requests per minute
• 10 requests per second

📘

You can find the API Usage Analytics and the API rate limits of your account in the Reports page.

Retrieving the Rate Limit State

Opsgenie provides the rate limiting state of each request via HTTP Headers. You can refer to the following table regarding the rate limiting headers:

Handling Throttles and Request Delays

When a request is rejected due to the rate limiting, you should retry the request after waiting for a reasonable time. Applying an exponential backoff algorithm rather than a simple retry logic provides better flow control for such scenarios. Exponential backoff will wait for progressively longer times between retries for request throttles.

The following pseudo-code is an example to retry sending a request using exponential delays between each attempt:

attemptCount := 0
shouldRetry := false

DO
    if attemptCount > 0
        Wait for 2^attemptCount x 200 milliseconds
    endif

    status := Result of the request

    IF status = THROTTLED
        shouldRetry := true
    ELSE IF status = SERVER ERROR
        shouldRetry := true
    ELSE
        shouldRetry := false
    END IF

    attemptCount := attemptCount + 1
WHILE shouldRetry AND attemptCount < MAX_NUMBER_OF_RETRIES

You can refer to the following references for further information about Exponential Backoff.

• Exponential Backoff and Jitter (AWS Architecture Blog)
• Exponential Backoff (Wikipedia)

Furthermore, please keep in mind that we came up with these rate limits for each domain after a year-long examination to make sure that our customers do not run into issues originating from that. We set the limits much more higher than the average number of requests of the accounts that send the most number of requests to Opsgenie, including the request bursts due to the outages or excessive monitoring events. Therefore, when you encounter throttled requests, it's always a good idea to review the configurations or the client setups on your side.