Building Resilient Systems: SRE Lessons from Telecom to Enterprise

Hope is Not a Strategy

Systems will fail. The only question is whether they fail when you are watching or at 3 AM on a Sunday. After running SRE for telecom platforms across 14 countries at Airtel and now managing 1,000+ Kubernetes clusters at Salesforce, here is what I have learned about building systems that survive failure.

Lesson 1: Define What "Reliable" Means Before Building

At Airtel, I inherited a platform with no formal SLOs. Engineers had an intuitive sense that "the system should be fast and available" but no shared definition of what that meant. Different teams had different expectations, and incidents were declared based on gut feeling.

The first thing I did was define SLOs for every critical service:

**Availability:** 99.95% measured at the load balancer level (not the pod level)

**Latency:** p99 < 500ms for API responses

**Error rate:** < 0.1% 5xx responses over any 5-minute window

This changed everything. Instead of arguing about whether an incident was "bad enough" to page someone, we had objective thresholds. Error budgets gave teams permission to ship faster when reliability was high and forced them to slow down when it was low.

Lesson 2: Multi-Region is Not Optional

At Airtel, we operated across 14 countries spanning Africa, South Asia, and the Middle East. Each region had different infrastructure constraints, latency requirements, and regulatory rules.

Key patterns that worked:

**Regional failover with global orchestration.** Each region runs autonomously but a central control plane manages traffic routing. If East Africa goes down, traffic shifts to the nearest healthy region.

**Data residency awareness.** KYC and subscriber data must stay in-country in many African markets. Our platform was designed from day one with data locality constraints.

**Degraded mode design.** When connectivity between regions is lost, each region continues serving requests from local cache. Full consistency is restored when connectivity returns.

Lesson 3: Chaos Engineering is Insurance, Not Luxury

We regularly inject failure into our systems:

At Salesforce (K8s Fleet Scale)

**Node termination:** Randomly kill nodes and verify workloads reschedule within SLO.

**Network partition:** Simulate cross-AZ network failures and verify service mesh handles graceful failover.

**Control plane stress:** Overload the API server and verify rate limiting and priority queuing work correctly.

**Certificate expiry simulation:** Advance time on test clusters to trigger cert rotation before real expiry.

At Airtel (Microservices Scale)

**Dependency failure injection:** Kill downstream services (SMS gateway, billing API) and verify upstream services degrade gracefully instead of cascading.

**Database failover testing:** Force primary-to-replica failover on PostgreSQL and verify zero-downtime switchover.

**Load spike simulation:** Replay peak traffic patterns (e.g., New Year's Eve SMS volumes) against staging to verify auto-scaling behavior.

The Result

Chaos engineering forced us to build systems that degrade gracefully. Instead of a hard crash, users experience slightly slower load times or reduced functionality. This is the difference between a P1 outage and a minor degradation that nobody notices.

Lesson 4: The Incident is the Easy Part

Resolving an incident takes hours. Preventing the *next* incident takes weeks. The post-incident review is where the real reliability work happens.

Our post-incident framework:

**Timeline reconstruction.** What happened, when, and in what order. No blame, just facts.

2. **Contributing factors.** Not "root cause" (complex systems rarely have a single root cause) but the set of conditions that made this incident possible.

3. **Action items with owners and deadlines.** Not "improve monitoring" but "add p99 latency alert for service X with threshold Y, owned by Z, due by [date]."

4. **Follow-through tracking.** Every action item is tracked in our incident management system. We review completion rates monthly. Incomplete action items from post-mortems are the #1 predictor of repeat incidents.

Lesson 5: Observability is Not Monitoring

Monitoring tells you *when* something is wrong. Observability tells you *why.* The distinction matters enormously at scale.

At Salesforce, we built a three-layer observability stack:

**Layer 1: Fleet health.** Are all 1,000+ clusters healthy? Which ones need attention? This is the "air traffic control" view.

**Layer 2: Cluster detail.** For a specific cluster, what is the node health, pod health, resource utilization, and recent change history?

**Layer 3: Workload detail.** For a specific application, what are the request rates, error rates, latency distributions, and dependency health?

The key insight: you need all three layers, and you need to navigate between them in seconds. When an alert fires, the engineer should go from "fleet overview" to "the specific pod on the specific node that is causing the issue" in under 60 seconds.

Lesson 6: Automation Compounds

Every manual operation you automate saves time forever. At scale, this compounds dramatically:

**Manual cert rotation:** 30 minutes per cluster x 1,000 clusters = 500 hours/year. Automated: 0 hours.

**Manual node replacement:** 45 minutes per node failure x ~200 failures/year = 150 hours/year. Automated: 0 hours.

**Manual incident triage:** 20 minutes per incident x thousands of alerts/month. With AI enrichment: 5 minutes per incident.

We track "toil hours saved" as a key SRE metric. Last quarter, our automation saved approximately 2,000 engineer-hours. That is an entire engineer's year of work, recovered through tooling.

The Takeaway

Reliability engineering is not about preventing all failures. It is about building systems where failures are expected, detected quickly, contained automatically, and learned from systematically. The tools change (Kubernetes today, something else tomorrow) but these principles are durable.