Service Level Objectives: Guide to Best Practices

Service level objectives (SLOs) help to measure system reliability. They define clear targets for system performance and a threshold below which the user experience will be considered degraded and the platform unstable.

This article will cover SLOs in-depth, along with related concepts like SLAs, SLIs, error budgets, burn rates, and time windows. We will also explore advanced topics like composite SLOs, SLO alerting strategies, common challenges, and some recommendations to help you optimize the use of SLOs.

Summary of service level objectives key concepts

The table below summarizes seven essential service level objective concepts this article will explore in detail. 

Service level objective (SLO)

Service level objectives (SLOs) are a set of targets, often expressed in percentages, defined by service providers and internal teams. They help ensure the expected performance and reliability delivered to users of a service. They are crucial in monitoring the service quality you offer and indicating whether you are meeting user expectations or are at risk of breaching agreements with the stakeholders.

For example, a team can define an SLO that specifies that an application must be available 99.5% of the time over one year. Another SLO could state that 97% of web page load time from users should be less than 3 seconds within one month. These measurable targets help align teams on what represents an acceptable performance and provide a benchmark for tracking progress over time.

Service level objectives have three components: Service, Level, and Objective 

Service level indicator (SLI)

SLIs are the foundation of service level objectives (SLOs), representing the metrics upon which they are built. Those metrics, such as availability, latency, or error rate, indicate your system's real-time performance.

While SLIs indicate the service's performance, the SLO defines the target for that performance.

To calculate the performance of a service or application at a point in time, you can use this formula:

Good Events / Total Events * 100

In this context, the SLI represents the metrics used in the queries to calculate "good" and "total" events. At the same time, the SLO defines the target of that calculation to determine whether the service meets the required performance.

Key elements to define SLI:

Three elements are important for defining SLIs: the type of service observed, the SLI metric, and the time window.

1. Type of services observed

Common service types include:

• Request-driven: Services where users send requests and expect responses, such as web browsers interacting with HTTP services or APIs for mobile apps.
• Pipeline: Processes that transform input data into a different output such as converting videos from one format to another or aggregating log files from multiple sources to create reports.
• Storage: Services that store data (e.g., files, records, or videos) for later retrieval.

2.  SLI metrics

The table below shows the standard SLI metrics for each type of service, with an SLO example for each SLI type

3. Time windows

Time windows define the period during which the error budget is calculated for a given SLO. There are two types of time windows:

• Rolling window: A continuously moving time window that doesn’t have fixed start and end dates. For example, for a rolling window of 30 days, we calculate the metric data from the past 30 days. It continuously evaluates the SLO over the last X days, weeks, or months.
  It is recommended that the rolling window be defined in weeks to ensure every rolling window has the same number of weekends. Generally, a rolling window of 4 weeks would be an excellent general-purpose time window.

• Calendar-aligned:  Calendar-aligned windows calculate the SLO in fixed periods, such as a month, a quarter, or a year. The error budget is restored at the beginning of each calendar window. For example, instead of a 30-day rolling window, you can calculate your metric for the first week of the last month.

Service level agreement (SLA)

A service level agreement (SLA) is a written contract between a service provider and a customer, created by the legal and business teams, that defines the expectations and commitments between both sides and the consequences for not meeting the agreed SLA.

While SLAs and SLOs are both reliability targets, they differ in purpose. SLAs typically have a legal remedy associated with them, such as monetary credits. Conversely, SLOs are internal targets that define measurable performance metrics. In another context, SLOs focus on technical performance goals, while SLAs provide the legal framework that formalizes those objectives.

In order to meet an SLA, teams should generally use an SLO target that is higher. For example, an SLA might include a term for 99.0% availability, while the service provider targets an SLO of 99.5% to provide a buffer before breaching the SLA. An easy way to tell the difference between an SLO and an SLA is to ask yourself, “What happens if the objective is unmet?” If there is no legal consequence, you are talking about an SLO.

SLA is a more relaxed target than SLO (Source)

Error budgets

Error budgets are the acceptable amount of non-compliance before the SLO is considered to be breached. Any downtime or errors will consume the error budget, and if the errors continue for a longer time, the error budget will be completely consumed, leading to violating the SLO. On the other hand, if the service operates perfectly, the error budget will remain intact.

Think of an error budget as your monthly household budget. You have money to cover essential expenses like rent, utilities, and groceries. This is similar to the reliability targets defined by your SLOs. If you’re staying within your budget, you can take risks, like buying a TV or decorating a room, similar to developers focusing on new features or innovations. However, if unexpected expenses arise and you’re close to overspending, you might need to skip the luxuries and focus on the critical needs; similar to the developers, when they are near consuming the error budget, they stop further development and apply code freezes to reduce any further risks.

The error budget is calculated with this formula:

Error Budget % = 100% - SLO %

For example, if your SLO is 99.5%, your error budget is 0.5% of the total duration or 0.5 % of the total expected events over a defined time period.

Error Budget = Error Budget % * Total time window or total number of events

The table of nines below might help you understand how an error budget is estimated from the SLO and that every extra nine of reliability you commit to in your SLO means a lower error budget value and, accordingly, a higher risk.

Burn rate

The burn rate is the rate at which the error budget is consumed. A 1x burn rate means that the error budget will last for the time window and be consumed at the end of it. A burn rate higher than 1x indicates that the error budget is burnt too quickly, which would result in breaching the SLO.

A burn rate higher than 1x will consume the error budget before the time window ends. (Source)

To calculate the burn rate, you need to be able to calculate the SLI's error rate during a specific time window. 

Error rate = Number of Failed Events / Total Number of Events * 100

Imagine you have a web application with 50 failed login attempts from a total of 2000 attempts during the last week.

Error rate = 50 / 2000 * 100 = 2.5%

Once you have calculated the error rate, you will be able to calculate the burn rate of your SLO with this formula:

Burn rate = Error rate in a time window / (100 – SLO%)

For the previous example, if the availability SLO of the web application is 95%:

Burn rate = 2.5 / (100 – 95) = 0.5 %

Composite SLO

In complex applications, several components are needed to deliver the final service for the user. While each component may have its own Service Level Objective (SLO), determining the overall reliability of the application requires more than just looking at those individual SLOs.

For example, imagine an application composed of three microservices: a database, an API, and a front-end service. Each of these has its own SLO, and the system will only operate if the three microservices are available. How can we calculate the SLO of the application itself?

This is where a composite SLO comes in. A composite SLO combines multiple SLO targets into a single SLO that represents your service's overall reliability.

Weighting in composite SLOs

Earlier, we introduced the concept of composite SLOs, which combine multiple SLOs into a single target. But what happens when an application consists of several components with different criticality levels, each having a unique SLO target? How can the composite target be evaluated to account for these variances?

Here comes the importance of weighting in composite SLOs. as it helps to ensure that the critical components in the application have more influence on the target than the other services by assigning a weight to each component SLO.

Use case 1

Let’s consider an application with four services, A, B, C, and D, with SLOs 99%, 98%, 95%, and 97 $, respectively. Here, we assign equal weights with a weight of 1 for each service.

Normalized weight = Weight of service / Sum of all weights * 100 = 1 / 4 * 100 = 25 %

In this case, the composite SLO can be set as follows:

Composite SLO = (99% * 25% + 98% * 25% + 95% * 25% + 97% * 25%) * 100 = (0.2475 + 0.245 + 0.2375+ 0.2425) * 100 = 97.52%

Use case 2

For the same previous example, we will assign different weights for services A, B, C, and D as 8,6,4 and 2, respectively.

In this case, the composite SLO can be set as follows:

Composite SLO = (99% * 40% + 98% * 30% + 95% * 20% + 97% * 10%) * 100 = (0.396 + 0.294 + 0.19+ 0.097) * 100 = 97.7%

How to assign weights

There's no universal rule about weighting individual SLOs; the best approach will depend on your system's context and specific requirements. Here are some bases on which you can assign the weight:

Based on SLO Type

• Availability-Critical SLOs:
  In most cases, service availability is more important than service response time, as a slow but functional service is better than a completely unavailable service. In such cases, it is recommended to assign a higher weight to the availability SLOs than the latency SLOs.
• Latency-Critical SLOs:
  In some specific services, such as online meetings and video conference tools, the response time will be as important as the service's availability.  In such cases, it is recommended that equal weights be assigned for availability and latency SLOs.
• Based on business requirements:
  When creating a composite SLO for a service or application that includes a multi-step user journey, it's recommended to assign weights based on the relative business importance of each step. The steps that have more influence on the business should have higher weights.
  For example, an online grocery application includes user journey steps like browsing categories, order placement, payment checkout, and feedback giving. For example, the ordering and payment steps would be more critical than the others, so they will have higher weights assigned to them.

Alerting on SLOs

An alert on SLO measures and notifies how much deviation from the SLO has occurred, ensuring that notifications are triggered well before the error budget is exhausted. This helps teams take preventive actions rather than reacting after the SLO has been violated.

According to Google's SRE books, there are some key factors to evaluate an SLO alerting system:

• Precision: The proportion of failure events detected. A higher rate indicates that every alert corresponds to an event.
• Recall: The percentage of failure events were detected. A higher percentage ensures failure events are not missed.
• Detection time: How quickly an alert was triggered after an issue starts. Short detection times minimize the consumption of the error budget and enable faster response.
• Reset time: How quickly an alert triggers if there's an issue and how quickly an alert clears after the issue is resolved, which helps the operations team focus on new incidents.

Alerting on the remaining error budget

Error budget-based alerts track the amount of the error budget consumed and the amount remaining. This method helps clarify how much room remains before an SLO breach.

There are two methods to alert on the remaining error budget:

• Alert on the remaining percentage: For example, an alert is triggered when the service has only 20% of its error budget left.
• Alert on the remaining duration: An alert will trigger when only 90 minutes of allowable downtime remain in the SLO window.

To calculate the error budget remaining, you must first calculate the error budget consumed:

Consumed Error Budget = Number of failed events / Number of the allowed to-fail events (Error budget) * 100

Remaining Error Budget = 100 % - Consumed Error Budget

Let’s think about an online store that promises an SLO of 99.5% availability over a 30-day time window. From the historical data, we can expect this online store to handle 100,000 order requests over the 30-day period, which translated into an error budget of 500 orders. During the month, 50 order requests failed to process.

Error Budget Consumed = 50 / 500 * 100 = 10 %

Remaining Error Budget = 100 % - 10 % = 90 %

Alerting on the burn rate

Alerting on burn rates helps to notify teams about failure incidents that cause an error budget to be consumed before the end of the SLO time window. 

Some may think they can ignore alerting and check the burn rate from time to time, but that’s not right; even the low and steady burn rates, which might go unnoticed,  can significantly consume the error budget in the long term of the time window.

To calculate the burn rate, we use this formula:

Burn rate = (Budget consumed × Compliance period) / Alerting window

For example, if a team decides that consuming 10% of the error budget (20 out of 200 allowed failures) in one hour is critical and should be alerted on, they can calculate the burn rate for a 30-day compliance period as follows:

Burn rate = 10% × 720 hrs / 1 hr = 72

The challenge here would be:

This burn rate alert will never alert if the error rate is lower than the burn rate of 72. A burn rate of 25 will still consume all the budget in 8 hours.

Time to consume the budget = Error Budget / Burn rate per hour = 200 / 25 = 8 hours

Multiple burn rate alerts

Multiple burn rate alerts ensure that the low and steady burn rates that will consume the error budget in the long term are detected and notified.

For example, we can alert in these cases:

• If the burn rate over the last 1 hour consumes 2% of the budget
• if the burn rate for the last 6 hours consumes 5% of the budget
• if the burn rate for the last 3 days consumes 10% of the budget

The challenge would be:

The long reset time for the alert. For example, in the last alert, you would wait 3 days to know that 10% of your error budget had already been consumed. After the team applied their code changes, you would wait another 3 days to evaluate the result. It will also require some mechanism of alert suppression so as to avoid alerts firing multiple times for the same events.

Best practice: Multi-window, multiple-burn rate alerts

This is the recommended alerting method in which an alert is configured on two different alerting windows: a short one and a long one; the shorter time window. This type of alert is for sure more complex than the previous methods. Still, it ensures that the internal team is notified in both cases when there’s a critical issue with a high burn rate and also notified when there is a steady and slow burn rate that might exhaust the burn rate in the long term.

For example, we can alert in these cases:

• If the burn rate over the last 1 hour consumes 2% of the budget and the burn rate over the last 5 minutes consumes 2% of the budget
• if the burn rate for the last 6 hours consumes 5% of the budget and the burn rate for the last 30 minutes consumes 5% of the budget
• if the burn rate for the last 3 days consumes 10% of the budget and the burn rate for the last 6 hours consumes 10% of the budget

Alert fatigue

Alert fatigue happens when an excessive number of alerts, often including false positives, are triggered, overwhelming the operations team. This leads to critical alerts being missed or delayed. False positives are alerts that were flagged to be exhausting the SLO but, after further analysis, were found not to be.

Symptom-based alert over a caused-based alert

In addition to multi-windows and multi-burn alerts, there are two common types of alerts teams should consider: cause-based and symptom-based. 

• Cause-based alerts: Focus on specific technical measurements. For example, you can alert your on-call team when the CPU usage over a database exceeds 90%, but that high CPU usage may not be meaningful to the user and might not affect your SLO budget at all. This will cause alert fatigue.

• Symptom-based alerts: Focus on specific symptoms, like latency. For example, you can alert your on-call team when 98% of the database queries take more than 500ms over the last 5 minutes. This is an incident that requires attention as it will breach the SLO.

Alert cooldown

Another way to overcome alert fatigue is the cooldown feature. Nobl9 provides this out-of-the-box feature to users to ensure that alerts are not repeatedly triggered for the same issue in a short period of time. Applying a cooldown period allows your team time to resolve the incident without being overwhelmed by duplicate alerts.

It is a parameter you define while creating an alert policy that establishes the amount of time that must pass to change the status of an alert from the okay status to an alert being triggered or from an active alert to the resolved status.

A lifecycle of an alert in Noble9 configured with a cooldown time. (Source)