When Retries Make Things Worse: Lessons from the GitHub Outage

Distributed systems often fail in unexpected ways.

Sometimes it's a database outage. Sometimes it's a networking issue. Sometimes a dependency becomes unavailable.

However, some of the most interesting failures happen when clients behave exactly as they were designed to.

A recent GitHub outage is a great example of this.

The Incident

GitHub experienced an outage where a portion of API requests started receiving 401 Unauthorized responses.

Under normal circumstances, a 401 response usually indicates an expired or invalid token.

Most applications handle this automatically:

text
401 Unauthorized
↓
Refresh Token
↓
Retry Request

This is the correct behavior.

However, the issue wasn't an expired token.

The credentials were valid, but the authentication layer was incorrectly returning 401 Unauthorized responses for legitimate requests.

From the client's perspective, the response looked completely normal, which is what made the incident interesting.

When Correct Behavior Creates More Problems

Imagine thousands of applications communicating with GitHub.

Each application receives a 401 Unauthorized response.

Each application assumes its token has expired.

As a result, every application immediately attempts to refresh its token.

text
App 1 → Refresh Token
App 2 → Refresh Token
App 3 → Refresh Token
...
App 10000 → Refresh Token

At first glance, this seems reasonable.

After all, refreshing a token after receiving a 401 is exactly what the client is supposed to do.

The problem is that all clients are doing it simultaneously.

Now the authentication infrastructure must handle:

• →Original API requests
• →Token refresh requests
• →Retried requests

What started as an authentication issue quickly becomes a traffic problem.

Understanding Retry Storms

Retries are one of the most common reliability techniques in distributed systems.

They work well for:

• →Temporary network failures
• →Short-lived service disruptions
• →Transient timeouts

However, retries can become harmful when the dependency itself is struggling.

More failures lead to more retries.

More retries create additional load.

Additional load leads to more failures.

text
Failure
↓
Retry
↓
More Load
↓
More Failures
↓
More Retries

This feedback loop is known as a Retry Storm.

A small issue can quickly grow into a much larger outage if enough clients continuously retry failed operations.

The Problem With Blind Retries

Consider the following logic:

javascript
while (response.status === 401) {
  refreshToken();
  retryRequest();
}

The logic appears reasonable.

However, if the authentication service itself is unhealthy, the application enters a continuous cycle:

text
401
↓
Refresh Token
↓
401
↓
Refresh Token
↓
401
↓
Refresh Token

One application doing this is unlikely to cause problems.

Thousands of applications doing it simultaneously is a different story.

Instead of helping the service recover, clients continuously add more work to an already overloaded system.

Exponential Backoff

A better approach is to slow down retries after repeated failures.

For example:

text
Retry #1 → Wait 1 second
Retry #2 → Wait 2 seconds
Retry #3 → Wait 4 seconds
Retry #4 → Wait 8 seconds

This technique is called Exponential Backoff.

Rather than repeatedly sending requests as fast as possible, clients gradually increase the delay between retry attempts.

This reduces pressure on the dependency and gives it time to recover.

Circuit Breakers

Sometimes even exponential backoff isn't enough.

At some point, the application should recognize that the issue may not be temporary.

This is where the Circuit Breaker Pattern becomes useful.

text
Failure
↓
Failure
↓
Failure
↓
Circuit Opens
↓
Stop Requests
↓
Wait
↓
Try Again Later

Instead of continuously retrying, the application temporarily stops sending requests and allows the dependency time to recover.

A circuit breaker prevents a failing service from being overwhelmed by additional traffic during an incident.

Building Fault-Tolerant Systems

One of the key lessons from this outage is that reliability isn't simply about adding retries.

Reliable systems are designed to fail gracefully.

Modern distributed systems commonly use:

• →Retries
• →Exponential Backoff
• →Circuit Breakers
• →Timeouts
• →Rate Limiting
• →Monitoring and Alerting

Together, these patterns help prevent small failures from becoming large-scale outages.

Final Thought

The interesting part of this outage wasn't the authentication failure.

It was the fact that thousands of applications did exactly what they were supposed to do and still made the situation worse.

In distributed systems, the challenge isn't just handling failures.

It's making sure your recovery strategy doesn't become the next failure.