API Rate Limiting
To protect the system from being overwhelmed, to provide more stable throughput
and to provide fairness among different accounts in terms of resource consuming,
OpsGenie applies rate limiting to the API and integration requests. If a request
exceeds the defined rate limit, the request is either rejected with the HTTP status
429 - Too Many Requests or delayed to be executed.
- Rate Limiting Strategy
- Rate Limit Calculation
- Retrieving the Rate Limit State
- Handling Throttles and Request Delays
Rate Limiting Strategy
Token Bucket Algorithm is used for rate limiting, where each received request corresponds to a single token. The limit of the request rate, the size of the bucket and the rate of adding a new token, is defined according to the following factors:
- OpsGenie splits received requests into 4 categories in terms of request domains:
- Alert: It's the domain for any request to Alert Rest API or Alert API v1 endpoints.
- Heartbeat: It's the domain for any request to Heartbeat Rest API or Heartbeat API v1 endpoints.
- Integration: It's the domain for any request except the ones to Rest API and API v1 endpoints. Please note: received e-mails via Email Integrations are also counted as Integration domain.
- Configuration: It's the domain for any request to the remaining Rest API or API v1 endpoints like Team API, User API, Integration API, Escalation API, etc.
- OpsGenie checks the rate limit status of any request to any domain in terms of
two different time interval units:
- Time unit interval types (that are minutes and seconds) form a separate bucket for each request domain according to the Token Bucket Algorithm and there are two buckets for each domain as a matter of course. When a request for a domain exceeds the rate limit of at least one of the related buckets, the request is either throttled or delayed.
- To summarize, when OpsGenie receives a request, the limits for both minute rate and second rate are checked first. If either of the limits are exceeded, the request is throttled (or delayed).
There are 3 possible rate-limiting states for each request:
|Allowed||The request does not exceed any of the defined rate limits. In this case, the request will be processed with its normal flow.|
|Delayed||OpsGenie allows the requests that do not exceed the rate limit more than a defined threshold by delaying them for 5 seconds instead of rejecting the requests. Please note: that delaying a request is available only for the requests to Rest API endpoints. OpsGenie throttles the requests to other endpoints in this case.|
|Throttled||The request is rejected and OpsGenie responds with the HTTP status
Rate Limit Calculation
The account-wide rate limit of each domain is calculated as follows:
- Limit per Minute:
BaseLimit* + (#OfUsers** x 4.0)
- Limit per Second:
BaseLimit* + (#OfUsers** x 0.067)
- * BaseLimit is the pre-defined number of allowed requests to the
related domain for the time interval. Base limit is defined according to
the following factors:
- Time Interval Unit: The base limit of the number of requests per minute is higher than the base limit of the number of requests per second.
- Subscription Plan Type: Each subscription plan has a different minimum rate limit. Please note: that Enterprise plan has the highest minimum limit and free plans have the lowest minimum limit. You can refer here for further information about our subscription plans.
- Request Domain: The rate limits of the requests to Configuration domain are lower than the other 3 domains.
- ** #OfUsers is the number of purchased user seats in your account subscription.
For example, assume that your account is subscribed to Standard plan with 60 users. If the base limit of requests per minute is 500 and the base limit of requests per second is 100, the rate limits for your account would be as follows:
- 740 requests per minute
- 101 requests per second
- 74 requests per minute
- 10 requests per second
Retrieving the Rate Limit State
OpsGenie provides the rate limiting state of each request via HTTP Headers. You can refer to the following table regarding the rate limiting headers:
|Header Name||Description||Header Value|
|X-RateLimit-State||Shows the rate limiting state of the request. This header is provided for the responses of all requests, no matter what the rate limit state is.||
|X-RateLimit-Reason||Shows the reason of the rejection (or delay). This header is provided only if the request is rejected or delayed.||
|X-RateLimit-Period-In-Sec||Integer value that represents the time interval of the rate limit that was exceeded. This header is provided only if the request is rejected or delayed, and the value is provided as seconds.||
Handling Throttles and Request Delays
When a request is rejected due to the rate limiting, you should retry the request after waiting for a reasonable time. Applying an exponential backoff algorithm rather than a simple retry logic provides better flow control for such scenarios. Exponential backoff will wait for progressively longer times between retries for request throttles.
The following pseudo-code is an example to retry sending a request using exponential delays between each attempt:
attemptCount := 0 shouldRetry := false DO if attemptCount > 0 Wait for 2attemptCount x 200 milliseconds endif status := Result of the request IF status = THROTTLED shouldRetry := true ELSE IF status = SERVER ERROR shouldRetry := true ELSE shouldRetry := false END IF attemptCount := attemptCount + 1 WHILE shouldRetry AND attemptCount < MAX_NUMBER_OF_RETRIES
You can refer to the following references for further information about Exponential Backoff.
Furthermore, please keep in mind that we came up with these rate limits for each domain after a year-long examination to make sure that our customers do not run into issues originating from that. We set the limits much more higher than the average number of requests of the accounts that send the most number of requests to OpsGenie, including the request bursts due to the outages or excessive monitoring events. Therefore, when you encounter throttled requests, it's always a good idea to review the configurations or the client setups on your side.