To protect the system from being overwhelmed, to provide more stable throughput and to provide fairness among different accounts in terms of resource consuming, Opsgenie applies rate limiting to the API and integration requests. If a request exceeds the defined rate limit, the request is either rejected with the HTTP
status 429 - Too Many Requests or delayed to be executed.
Token Bucket Algorithm is used for rate limiting, where each received request corresponds to a single token. The limit of the request rate, the size of the bucket and the rate of adding a new token, is defined according to the following factors:
Opsgenie splits received requests into 4 categories in terms of request domains:
- Alert: It's the domain for any request to Alerts or Alert API v1 endpoints.
- Heartbeat: It's the domain for any request to Heartbeat Rest API or Heartbeat API v1 endpoints.
- Integration: It's the domain for any request except the ones to Rest API and API v1 endpoints (Web API deprecated).
- Configuration: It's the domain for any request to the remaining Rest API or API v1 endpoints (Web API deprecated) like Team API, User API, Integration API, Escalation API, etc.
- Incident: It's the domain for any request to Incident API endpoints.
- Email: It's the domain for any email received via Email Integrations or Heartbeats.
- Search: It's the domain for any API request made to the list alerts and users.
- Schedule-oncalls: It's the domain for any request made to retrieve the current and next on-call user(s).
Please note: that the rate limiting is applied separately for each request domain. In other words, exceeding the rate limit for a domain doesn't mean that the rate limit is exceeded for other domain(s).
Opsgenie checks the rate limit status of any request to any domain in terms of two different time interval units:
When a request is received, Opsgenie decides the request status by checking the allowed number of requests per minute and the allowed number of requests per second separately. Please note: that the time interval is tracked as a rolling window.
Time unit interval types (that are minutes and seconds) form a separate bucket for each request domain according to the Token Bucket Algorithm and there are two buckets for each domain as a matter of course. When a request for a domain exceeds the rate limit of at least one of the related buckets, the request is either throttled or delayed.
To summarize, when Opsgenie receives a request, the limits for both minute rate and second rate are checked first. If either of the limits are exceeded, the request is throttled (or delayed).
The request rate limits are account-wide and the requests from any integration of your account share the same limits by default. However, Opsgenie can apply integration fairness that will prevent each integration from consuming a defined percentage of the account-wide rate limits. You can contact us to activate integration fairness for your account.
There are 3 possible rate-limiting states for each request:
The request does not exceed any of the defined rate limits. In this case, the request will be processed with its normal flow.
Opsgenie allows the requests that do not exceed the rate limit more than a defined threshold by delaying them for 5 seconds instead of rejecting the requests. Please note: that delaying a request is available only for the requests to Rest API endpoints. Opsgenie throttles the requests to other endpoints in this case.
The request is rejected and Opsgenie responds with the HTTP status
BaseLimit is the pre-defined number of allowed requests to the related domain for the time interval. Base limit is defined according to the following factors:
- Time Interval Unit: The base limit of the number of requests per minute is higher than the base limit of the number of requests per second.
- Subscription Plan Type: Each subscription plan has a different minimum rate limit. Please note: that Enterprise plan has the highest minimum limit and free plans have the lowest minimum limit. You can refer here for further information about our subscription plans.
- Request Domain: The rate limits of the requests to Configuration domain are lower than the other 3 domains.
(#OfUsers is the number of purchased user seats in your account subscription.)
For example, assume that your account is subscribed to Standard plan with 60 users. If the base limit of requests per minute is 500 and the base limit of requests per second is 100, the rate limits for your account would be as follows:
- 740 requests per minute
- 101 requests per second
Additionally, if integration fairness is activated with 10% rate for your account (upon your request), the rate limit for each integration would be as follows:
- 74 requests per minute
- 10 requests per second
You can find the API Usage Analytics and the API rate limits of your account in the Reports page.
Opsgenie provides the rate limiting state of each request via HTTP Headers. You can refer to the following table regarding the rate limiting headers:
Shows the rate limiting state of the request. This header is provided for the responses of all requests, no matter what the rate limit state is.
OK: The rate limits are not exceeded and the request is processed with its normal flow.
BURST: The rate limits are exceeded with an acceptable limit. Request is accepted but processing is delayed for 5 seconds.
THROTTLED: The rate limits are exceeded and the request is rejected.
Shows the reason of the rejection (or delay). This header is provided only if the request is rejected or delayed.
ACCOUNT: You've exceeded your account-wide rate limits.
INTEGRATION: Integration fairness was activated for your account and the authorized integration of the request has exceeded its limit.
Integer value that represents the time interval of the rate limit that was exceeded. This header is provided only if the request is rejected or delayed, and the value is provided as seconds.
60: You've exceeded the limit of the number of requests per minute.
1: You've exceeded the limit of the number of requests per second.
When a request is rejected due to the rate limiting, you should retry the request after waiting for a reasonable time. Applying an exponential backoff algorithm rather than a simple retry logic provides better flow control for such scenarios. Exponential backoff will wait for progressively longer times between retries for request throttles.
The following pseudo-code is an example to retry sending a request using exponential delays between each attempt:
attemptCount := 0 shouldRetry := false DO if attemptCount > 0 Wait for 2attemptCount x 200 milliseconds endif status := Result of the request IF status = THROTTLED shouldRetry := true ELSE IF status = SERVER ERROR shouldRetry := true ELSE shouldRetry := false END IF attemptCount := attemptCount + 1 WHILE shouldRetry AND attemptCount < MAX_NUMBER_OF_RETRIES
You can refer to the following references for further information about Exponential Backoff.
Furthermore, please keep in mind that we came up with these rate limits for each domain after a year-long examination to make sure that our customers do not run into issues originating from that. We set the limits much more higher than the average number of requests of the accounts that send the most number of requests to Opsgenie, including the request bursts due to the outages or excessive monitoring events. Therefore, when you encounter throttled requests, it's always a good idea to review the configurations or the client setups on your side.
Updated 2 years ago