Best Practices for Creating a Modern IT Incident Management Program

Implement comprehensive observability tools

Monolithic applications based on a client-server architecture are easier to monitor because they operate as tightly coupled systems - processes share memory space, network latency is minimal, dependencies remain stable, and root causes can be traced directly. Modern applications, on the other hand, process transactions through interconnected services, including microservices hosted on ephemeral containers and services provided by third-party APIs. The infrastructure is configured as code (IaC), which creates abstractions that provide significant productivity improvements at the expense of new cascading failure modes. A delayed response or failed connection anywhere in this chain can trigger a domino effect affecting customer experience.

Effective detection requires monitoring every component involved in processing customer transactions. Frontend services generate early warning signals through page load times and API response metrics. A spike in client-side error might indicate problems with the product catalog service while increasing API latency could signal database issues.

This complexity gave rise to the term “observability” in recent years, in contrast with the term “monitoring” used for IT systems associated with client-server environments. The term observability originated from control theory and evolved in the context of software systems to emphasize gaining insights via four types of signals: Metrics, Events, Logs, and Traces, also known as MELT.

Before digging into the MELT components, it’s worth noting the OpenTelemetry project, which was created in 2019 through the merger of two open-source observability projects, OpenTracing and OpenCensus, to provide a unified framework for collecting telemetry data like traces, metrics, and logs. Now governed by the Cloud Native Computing Foundation (CNCF), it has become a leading reference framework for the observability of modern distributed systems. It is supported by over forty observability vendors, including some of the open-source tools we will reference in this article. 

eBPF-based monitoring is also rapidly gaining in popularity as an alternative, or enhancement to open telemetry. eBPF enables collecting observability data by allowing sandboxed programs to run within the operating system’s kernel. 

Metrics

Metrics provide quantitative measurements of system behavior over time. For example, an e-commerce platform would typically collect metrics like orders processed, payment processing times, and inventory updates at regular intervals, which may be once a second, minute, hour, or any rate in between depending on the application requirement, as long as it’s based on a regular interval. 

These measurements, known as time-series data, are used to establish normal operating patterns and help detect deviations. For instance, a sudden drop in order processing rate might indicate an issue with the checkout service, while increasing payment processing times could signal problems with the payment gateway. Time series metrics include measurements of key performance indicators from the infrastructure components like a database query response time, network latency, disk I/O, or the available storage space on a disk. 

The example below shows a comma-separated (CSV) time-series data sample capturing value for key metrics, such as CPU, memory usage, and network measurements, collected from a container every minute. Each row starts with a time stamp. 

timestamp,container_id,cpu_usage_percent,memory_usage_mb,network_in_kb,network_out_kb
2024-12-16T10:00:00Z,container_12345,12.5,256,1024,512
2024-12-16T10:01:00Z,container_12345,14.0,260,1100,540
2024-12-16T10:02:00Z,container_12345,13.8,262,1080,530
2024-12-16T10:03:00Z,container_12345,15.2,265,1150,560
2024-12-16T10:04:00Z,container_12345,12.9,259,1050,520
          

Prometheus is a popular free and open-source tool with integrations, known as exporters, for collecting metric data from commonly used infrastructure technologies.

Events

Events, on the other hand, capture significant changes in the system state that require attention. When a payment gateway fails to backup, inventory drops below acceptable levels, or a critical service restarts, these state changes generate events. Unlike time-series metrics, which establish patterns at a regular time interval, events mark discrete occurrences that may require immediate action. For example, a shopping cart service restart during peak hours demands investigation even if order metrics appear normal.

Logs

Logs provide a detailed history of system operations through timestamped records. While metrics might show that payment processing has slowed compared to usual, logs reveal the underlying activities that led to that point, helping the operations team troubleshoot. 

For example, application logs include user actions and errors, server logs capture web and database activity, and system logs record operating system events. Each component in the infrastructure typically has a dedicated log file, which is why it’s important to use tools like Elastisearch to aggregate and index logs so that they can be easily searched during troubleshooting.

The following is an example of a JSON log:

{
    "timestamp": "2024-12-14T10:15:30Z",
    "service": "payment",
    "error": "gateway_timeout",
    "order_id": "12345",
    "details": "Payment gateway unresponsive after 5s"
}
      

Traces

Traces map the request's journey through distributed services. A single checkout operation travels through multiple services—from shopping cart validation to payment processing to order creation. Traces show how long each step took as the transaction crossed application tiers like web and database servers, which helps pinpoint the bottlenecks. For example, if customers report slow checkouts, traces help identify whether the delay occurs in inventory checks, payment processing, or order creation.

Staying with our e-commerce platform example, the following Python method uses OpenTelemetry’s tracing API to monitor a checkout endpoint. OpenTelemetry provides a vendor-agnostic way to collect telemetry data that can be visualized in tools like Jaeger or Zipkin:

def process_checkout(cart_id, payment_info):
    with trace.span("checkout_operation") as span:
        span.set_attribute("cart_id", cart_id)
                   
        validate_cart(cart_id)
        check_inventory(cart_id)
        process_payment(payment_info)
        create_order(cart_id)
          

This method tracks the entire checkout flow, allowing teams to identify bottlenecks and failures at each step. 

One of the most popular open-source tools used for instrumenting applications with distributed tracing is Jaeger.

While events, logs, and traces serve different purposes in monitoring, they all fundamentally record system activity as timestamped entries. The key difference lies in how this data is structured and used - events trigger immediate actions, logs enable detailed troubleshooting, and traces help understand service dependencies and performance bottlenecks.

Implementing alerting thresholds

Alerting thresholds translate business requirements into measurable upper or lower limits. For an e-commerce platform, these limits often map directly to customer experience - how long will customers wait for a page to load? How many failed checkouts drive them to competitors? 

Start with baseline metrics that reflect normal operation. During non-peak hours, an e-commerce site might process 50 orders per minute with 2-second checkout times. During sales events, these numbers might jump to 500 orders per minute. Alerting thresholds must account for these variations to avoid false alarms while catching real issues. 

Common e-commerce thresholds include response time and error rate thresholds:

Translating these thresholds into Prometheus alerting rules would look like the following:

# Payment processing alerts
- alert: PaymentProcessingTime
  expr: payment_processing_duration_seconds > 3
  for: 5m
  labels:
    severity: warning
    service: checkout
        
# Order processing errors
- alert: HighOrderErrors
  expr: rate(order_errors_total[5m]) / rate(orders_total[5m]) > 0.01
  labels:
    severity: critical
    service: orders
      

These rules use PromQL (Prometheus Query Language) to define conditions that trigger alerts through Prometheus Alertmanager.

Most e-commerce platforms use a combination of application instrumentation and infrastructure monitoring. 

Application monitoring captures business-specific metrics using instrumentation libraries. For example, a payment processing method can be instrumented with Prometheus’s Python client library to record gateway timeouts like this:

def process_payment(order_details):
    start = time.time()
    try:
        result = payment_gateway.charge(order_details)
        duration = time.time() - start
        record_metric('payment_duration', duration)
        return result
    except GatewayTimeout:
        record_error('payment_timeout')
        raise
      

Infrastructure monitoring tracks system resources and service health. Some key monitoring indicators include: 

• Connection pool utilization, query latency deadlocks, and long-running transactions, for database health. 
• Queue depth, consumer lags, and message processing errors, for message queues. 
• Hit rates, eviction rates, and memory usage, for cache performance.

Detection systems should also adapt to changing conditions. During Black Friday sales, normal traffic patterns shift dramatically. Detection rules in Prometheus can use PromQL time-based functions to modify thresholds for peak periods:

# Dynamic threshold based on time of day
- alert: HighLatency
    expr: response_time_seconds > day_of_week == "Friday" ? 5 : 2
    labels:
        severity: warning
          

Most importantly, detection systems should focus on customer impact. A database running at 80% CPU might not matter if customers can still complete purchases quickly. However, a 1% increase in checkout errors directly affects revenue and requires immediate attention.

While these thresholds serve as useful indicators (SLIs), combining them into Service Level Objectives (SLOs) provides a more comprehensive view of system health. Instead of reacting to individual threshold breaches, teams can focus on maintaining overall service quality that aligns with user experience

Define service level objectives

Service Level Objectives (SLOs) transform raw monitoring data into measures of user experience, focusing on what matters to customers rather than just system metrics. For an e-commerce platform, for example, monitoring individual components provides the required data, but SLOs transform this data into measures of service quality that matter to customers and the business. While application servers might report 95% CPU utilization and databases show 1000 queries per second, customers care about whether they can complete their purchases quickly and reliably.  

Understanding basic SLO components 

Staying with our e-commerce platform example, a checkout completion SLO would demonstrate how metrics combine into meaningful measurements. When an SLO states that 99.9% of checkout attempts must be completed within 3 seconds, it requires monitoring and correlating data from multiple systems:

• Frontend servers measure initial page load and API response times
• Shopping cart services track item addition and price calculation speed
• Payment gateways report transaction processing times
• Inventory systems monitor stock check accuracy
• Order processing tracks confirmation delivery

Setting accurate SLOs depends on historical data analysis and sophisticated calculations. A platform processing millions of transactions generates patterns that help predict degradation and set appropriate thresholds. These patterns also expose unexpected relationships between components - for instance, operations teams might discover that slow product image loading times (due to high traffic and not enough hosting resources) consistently precede checkout slowdowns by a few minutes, which can be a leading indicator to scale resources preemptively.

Composite SLOs

While most monitoring tools can track simple SLOs, complex application platforms with multiple interconnected services must rely on composite SLOs. However, the technology behind composite SLOs has evolved significantly in recent years. 

A traditional composite SLO combines metrics from different sources to provide a complete picture of service health. For instance, an e-commerce checkout workflow must process data from payment gateways, inventory systems, and user sessions.

However, traditional composite SLO implementations face significant limitations. They often restrict data sources to a single project, support only a single level of hierarchies, and treat all components equally regardless of their business importance. These limitations can lead to missed patterns or unreliable results when dealing with distributed systems and varying traffic patterns.

Modern composite SLOs 

Modern composite SLOs (you can think of them as the 2.0 version of composite SLOs) overcome these limitations. They provide insights into complex systems from a single entry point down to the smallest component, helping teams have complete visibility of system reliability. Operations teams can combine many components per SLO across different data sources and projects, creating multi-level hierarchies that reflect their service architecture.

Consider this multi-level composite SLO for an e-commerce checkout flow (using Nobl9’s sloctl’s API):

apiVersion: n9/v1alpha
kind: SLO
metadata:
  name: checkout-composite
  displayName: Checkout Flow Reliability
  project: ecommerce
  labels:
    key:
      - service
      - tier
    value:
      - checkout
      - critical
spec:
  description: Composite SLO for the entire checkout process flow
  alertPolicies:
    - checkout-degradation
    - payment-failure
  budgetingMethod: Occurrences
  objectives:
    - displayName: Checkout Experience
      name: checkout-objective
      target: 0.99
      composite:
        maxDelay: 5m
        components:
          objectives:
            - project: frontend-monitoring
              slo: user-experience
              objective: frontend-latency
              displayName: Frontend Performance
              weight: 0.4
              whenDelayed: CountAsGood
            - project: payment-system
              slo: transaction-processing
              objective: payment-success
              displayName: Payment Processing
              weight: 0.6
              whenDelayed: CountAsBad
            - project: inventory
              slo: stock-management
              objective: inventory-accuracy
              displayName: Inventory Accuracy
              weight: 0.5
              whenDelayed: Ignore
  service: checkout-service
  timeWindows:
    - unit: Day
      count: 28
      isRolling: true
      calendar:
        timeZone: UTC
      

This hierarchical structure enables sophisticated reliability monitoring through a properly defined YAML configuration. Each component in the composite SLO carries a weight that reflects its business importance. In this example, payment processing (weight 0.6) has the highest priority, followed by inventory accuracy (0.5) and frontend performance (0.4). These weights help teams focus their attention on the system's critical parts. 

The configuration handles real-world complexity through several mechanisms. The `maxDelay` setting of 5 minutes provides a buffer for data collection delays, while the `whenDelayed` parameter for each component determines how to handle missing data. Frontend metrics count as successful when delayed (CountAsGood), payment processing failures count against the SLO (CountAsBad), and delayed inventory data is ignored. This flexibility maintains accurate reliability measurements even when monitoring systems experience temporary issues. 

The SLO configuration also includes practical operational elements. Alert policies trigger different responses for general checkout degradation versus payment failures. The 28-day rolling window provides a balanced view of system reliability, while labels help categorize and filter SLOs across the organization. 

For this purpose, organizations increasingly turn to specialized platforms that can: 

• Handle complex multi-service dependencies and integrations with all the leading monitoring tools 
• Maintain calculation accuracy at scale using advanced algorithms and adapt to changing traffic patterns.

Combining composite SLOs with the replay functionality helps users create advanced SLOs and test them in minutes instead of waiting weeks for fine-tuning. 

Leading-edge SLO solutions allow users to replay months of historical data in fast-forward mode in minutes to determine the appropriate SLO configuration, saving weeks of trial and error. You can learn more about the SLO replay functionality on this page.

Managing error budgets with composite SLOs

Error budgets translate SLOs into practical operational guidance. When setting a 99.9% uptime SLO, teams accept that 0.1% downtime is acceptable—this translates to 43.2 minutes per month. This error budget becomes a spending account for reliability, consumed through planned maintenance windows, feature deployments, infrastructure updates, and unplanned outages.

Modern composite SLOs enhance error budget management by maintaining historical context through configuration changes. Unlike basic implementations where editing components reset the error budget, teams can refine their reliability targets continuously without losing historical data.

For example, during the holiday shopping season, teams might adjust component weights to reflect changed business priorities - giving payment processing accuracy higher precedence over frontend performance metrics. These adjustments can be made without resetting error budgets or losing trending data

When error budget spending accelerates, teams implement reliability measures:

• Pause non-critical deployments
• Scale infrastructure proactively
• Postpone feature launches
• Focus on stability improvements

Decisions are based on actual system behavior rather than individual component metrics. 

Establish incident response procedures

When a platform experiences issues, every minute of downtime impacts revenue. Response procedures determine how quickly teams detect, assess, and resolve incidents. These procedures must define who responds, how they communicate, and what actions they take.

Define roles and responsibilities

Response effectiveness depends on role definition and responsibilities. Each incident involves multiple team members working in coordination. The following table captures the essence of the roles and responsibilities that need to be defined.

Creating escalation paths

Incidents follow a severity-based escalation path that determines response timing and team involvement. A rather simple escalation matrix for three severity levels would look like this:

However, incidents are dynamic events that can escalate rapidly. While initial severity assessment helps mobilize the appropriate response, teams must continuously evaluate the impact of the incident and adjust their response accordingly. Teams should also be able to make judgement calls on the severity of an incident based on the information available rather than being pigeonholed into a rigid definition.

Consider a product search slowdown initially classified as SEV3. The local team begins an investigation, expecting to resolve it within the standard 4-hour window. However, monitoring reveals that the search latency affects the product recommendation engine, impacting the checkout process. As customer complaints increase and revenue metrics show impact, the incident commander escalates the incident to SEV2, bringing in the technical lead and relevant SMEs. 

If the issue spreads, affecting core business functions like checkout completion, the incident commander might escalate to SEV1, triggering executive notification and full team response. A progressive escalation like this aligns response efforts with the incident's actual business impact. 

The following diagram illustrates this process:

Key escalation triggers include:

• Expanding the scope of affected systems
• Increasing customer impact
• Revenue implications
• Recovery time extending beyond initial estimates
• Dependencies affecting critical services

The incident commander reviews these key factors throughout the response effort and reevaluates severity levels and team engagement as needed. With a flexible approach, operations teams can deploy the necessary resources as their understanding of the incident increases.

Response playbooks

Since the terms playbook and runbook are often used interchangeably, let us first set context around our usage of the terms before we explain the concepts in more detail.

While playbooks and runbooks share similarities, they serve distinct purposes. Playbooks have a broader scope, often involving multiple teams or stakeholders, while runbooks focus on detailed, step-by-step instructions for specific procedures, such as rolling back a deployment.

Playbooks can transform the incident response from desperate scrambling to systematic action-taking when adopted properly. Consider this playbook for a payment system failure; from initial assessment to recovery verification, each milestone comes with a specific set of steps that need to be undertaken, depending on the nature of the problem: 

1. Initial assessment → Verify payment gateway status → Check database connectivity → Review recent deployments 
2. Stakeholder communication → Update status page with incident details → Notify customer service teams → Brief executive team if SEV1 
3. Resolution steps → Follow gateway provider's incident procedures → Document all system changes → Prepare for potential rollback 
4. Recovery verification → Monitor transaction success rates → Verify customer impact resolution → Document resolution steps

Communication protocols

Communication during incidents follows established patterns that prevent confusion and speed up incident resolution. There are two main types of communication: internal and external.

Internal communications maintain team coordination through dedicated incident response chat rooms,  regular status updates based on severity, handoff procedures between teams or team members, and documented decision points.

External communications keep stakeholders informed via regular status updates, support team briefings, executive summaries, and customer notifications. 

The industry standard for external communication is using status pages, such as this one: https://status.slack.com/. 

Documentation requirements

Every incident should generate documentation that serves current needs and future improvements. Items that typically need to be documented are:

• The timeline of key events
• The actions taken and their results
• Communication logs
• Any system changes that were implemented

This centrally stored documentation provides the foundation for future automation opportunities and post-incident analysis, both of which will be discussed in the upcoming sections of this article.

Automate incident resolution processes

When responding to incidents, consistent execution and speed matter more than heroic efforts. Automation removes human error from repetitive response tasks and enables rapid recovery, particularly for common failure scenarios that teams have previously resolved. 

Common automation scenarios

Staying with our e-commerce example, payment processing issues often follow predictable patterns with known resolutions. For instance, when a payment gateway becomes unresponsive, the resolution might involve switching to a backup gateway, restarting processing services, scaling up resources, or rolling back recent changes. 

Rather than manually executing these steps, teams can automate the response sequence using infrastructure as code tools. Ansible excels at orchestrating operational tasks, while Terraform manages infrastructure state changes. 

An Ansible script that automates payment gateway failover would look something like this:

- name: Payment Gateway Failover
  hosts: payment_servers
  tasks:
    - name: Check primary gateway health
      uri:
        url: ""
        return_content: yes
    register: health_check
   
- name: Switch to backup gateway
  when: health_check.status != 200
  block:
    - name: Update gateway configuration
      template:
        src: gateway_config.j2
        dest: /etc/payment/config.yml
   
- name: Restart payment service
  service:
    name: payment-processor
    state: restarted
   
- name: Verify backup gateway
  wait_for:
    timeout: 30
  register: gateway_check
            

This script first checks the primary gateway's health. If it detects issues, it updates the configuration to use the backup gateway, restarts the payment service, and verifies its operation. This automated sequence executes in seconds, compared to minutes, with a manual intervention.

For infrastructure-level recovery, Terraform enables automated rollback capabilities through deployment circuit breakers. The following example shows how it’s done:

resource "aws_ecs_service" "payment_service" {
  name            = "payment-processor"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.payment.arn
  desired_count   = 2

  deployment_controller {
    type = "CODE_DEPLOY"
  }

  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }
}

resource "aws_cloudwatch_metric_alarm" "payment_errors" {
  alarm_name          = "payment-error-rate"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "2"
  metric_name         = "ErrorRate"
  namespace           = "AWS/Application"
  period              = "60"
  threshold           = "1"
  alarm_description   = "Payment error rate exceeded threshold"
  alarm_actions       = [aws_appautoscaling_policy.rollback.arn]
}
        

This Terraform configuration establishes an ECS service with automatic rollback capabilities. If the error rate exceeds the defined threshold, the system automatically reverts to the last known good state. The CloudWatch alarm monitors the application's error rate and triggers the rollback when necessary.

Progressive automation levels

Teams typically implement automation progressively. The following table breaks down the automation process by level:

Integration with monitoring

Automated responses depend on reliable monitoring data to trigger actions. Monitoring systems track service health metrics, error rates, and resource utilization, feeding this data into automation decision points. When metrics breach defined thresholds, the system evaluates compound conditions before selecting and executing the response playbook. For instance, a spike in payment errors and increased API latency might trigger a different automated response than payment errors alone.

Validation and safety

Automation power requires well-designed safeguards. Every automated procedure includes pre-execution validation to verify the system state and post-execution checks to confirm successful recovery. Rate limiting prevents cascading automated actions, while circuit breakers stop automation loops. By putting these mechanisms in place, you can rest assured that automated responses will help rather than aggravate system stability.

The automated process generates detailed logs of its actions and results, which helps teams analyze automation effectiveness and refine response procedures over time. When automated responses fail or produce unexpected results, these logs provide debugging information for improving the automation rules.

Conduct a post-incident analysis

Post-incident analysis transforms incidents from unwelcome disruptions into opportunities for improvement. A blameless post-mortem brings together stakeholders to understand what happened, why it happened, and how to prevent similar issues.

Define a post-mortem meeting format

The post-mortem meeting follows a structured format to make sure a comprehensive analysis is done. Participants first review the incident timeline, establishing when the issue was detected and what actions were taken. This chronological review often reveals gaps in detection or response that might otherwise go unnoticed. 

Impact analysis follows, examining both technical and business consequences. Teams assess which services degraded, how many customers were affected, and what revenue impact occurred to prioritize future preventive measures. 

The core of the meeting is root cause investigation. Teams examine system logs, configuration changes, and deployment records to identify contributing factors. The focus must remain on systemic issues rather than individual mistakes.

Consider this example from our ongoing case of a platform's payment system outage: 

• At 09:15, error rates began increasing.
• Automated alerts triggered five minutes later at 9:20
• The team began an investigation by 09:25
• They identified the root cause at 09:45
• A Terraform configuration change was deployed without proper validation
• System restoration completed by 10:15
• An all-clear was announced at 10:30

The investigation revealed several process improvements needed: 

Use templates for consistent documentation 

Post-incident documentation serves as both a historical record and an improvement guide. An incident report begins with an overview, capturing the incident's duration, severity, and impact on services and customers. Technical analysis details the triggering events, system behavior, and resolution steps. 

The response assessment examines how effectively teams detected and addressed the incident, which includes evaluating team coordination, communication flow, and the effectiveness of automated tools. Each assessment identifies opportunities for improving response procedures and automation. 

The following table lists example categories that should be included in an incident postmortem template:

You can access an incident retrospective template at this link (here) in Google Docs format to get started. 

Track action items 

Action items from post-mortems demand systematic tracking. Each improvement initiative needs an owner with decision authority and a realistic implementation timeline. Success criteria help teams measure progress, while dependency analysis prevents implementation bottlenecks. 

Project management systems track these items, linking them to specific incidents for context. Regular reviews ensure progress continues and identify blocked items needing escalation. This systematic approach transforms post-mortem findings into concrete improvements. 

Measure improvements

Post-incident analysis helps teams track how SLO violations affect system reliability over time. The analysis examines violation patterns: their frequency, duration, and severity. For instance, repeated violations of the checkout SLO during peak hours might indicate insufficient capacity planning, while sporadic violations across different components could point to systemic monitoring gaps.

These violation patterns guide improvement efforts. When a particular service consistently breaches its SLO, teams might need to adjust its architecture or strengthen its resilience. If violations occur mainly during deployments, teams might revise their deployment procedures or implement stricter pre-deployment testing. Regularly reviewing these patterns helps teams prioritize their reliability investments and validate the effectiveness of their improvements.

Consider our e-commerce platform's payment processing example; SLOs may show frequent violations during promotional events, and the team might implement automatic scaling policies triggered by traffic patterns. After implementing these improvements, the team can measure success by tracking the reduction in SLO violations during subsequent high-traffic periods.

Last thought: proactive IT incident management with Nobl9

While comprehensive incident management processes and tools form the foundation for handling issues effectively, the shift from reactive to proactive management faces several challenges. Most organizations need help connecting their monitoring data to meaningful reliability metrics and automating responses effectively. 

Traditional SLO management tools need help keeping pace as systems become more complex. Teams face increasing challenges integrating data from multiple sources, handling intricate service dependencies, and calculating accurate thresholds. Manual system data analysis can take weeks or months, delaying adjustments to SLO thresholds and potentially midding important patterns.

Nobl9, a founding pioneer of the OpenSLO project, offers a platform designed specifically to address these challenges.

Nobl9 transforms this process through unified reliability management. The Service Health Dashboard shows service degradation in context, while error budget tracking helps teams decide on response urgency. Historical analysis through SLI Analyzer reveals if similar patterns preceded past incidents, enabling preventive action. Automated reliability management then closes the loop between detection and response. 

Incident management is often a reactive process, but the transition from being reactive to proactive requires modern SLO tools like Nobl9 to tie infrastructure telemetry with measurable service quality, runbooks, and playbooks for systematic remediation, processes for escalation and post-mortem analysis, and finally, training of the operations and development staff on the tools and processes. The combination helps organizations adopt modern IT incident management techniques applicable to every application, whether running on dedicated servers or Kubernetes clusters.