When “Works on Lambda” Breaks on ECS: A Concurrency Bug from Misusing Singletons in .NET

We recently migrated a .NET Core application from .NET 8 to .NET 10 and moved its hosting model from AWS Lambda to ECS. The service acts as a wrapper around a downstream API that reads and updates client “fact find” data.

Post-migration, we started seeing critical issues:

• Users were seeing other users’ data
• Updates were being applied to the wrong clients
• Users could access data they shouldn’t have permission to see

This was a data isolation failure. The root cause turned out to be a misuse of a singleton combined with differences in execution models between Lambda and ECS.

---

Architecture Context

The service:

• Accepts user requests
• Adds headers (auth/user context)
• Calls a downstream API
• Returns processed responses

A shared configuration object (ConnectionOptions) was introduced to manage headers for outbound calls.

This object:

• Was registered as a singleton
• Was mutated per request to inject headers

---

Why It “Worked” on Lambda

AWS Lambda has a different execution model:

• Each invocation typically runs in isolation
• Even with warm starts, concurrency per instance is limited
• No true parallel execution within a single instance (by default)

So:

• Mutating a singleton per request appears safe
• Concurrency issues are masked

The flawed design went unnoticed.

---

What Changed in ECS

ECS runs your app as a long-lived service:

• Multiple requests are processed concurrently
• Threads share the same memory space
• Singleton services are shared across all requests

Example

public class ConnectionOptions
{
    public Dictionary<string, string> Headers { get; set; }
}

services.AddSingleton<ConnectionOptions>();

connectionOptions.Headers["Authorization"] = userToken;

---

The Problem

Two requests arrive at the same time:

Request A	Request B
Sets header to User A	Sets header to User B
Calls downstream API	Calls downstream API

Because both share the same singleton:

• Headers overwrite each other
• Requests leak state
• Downstream API gets incorrect user context

This leads to:

• Cross-user data exposure
• Incorrect updates
• Non-deterministic failures

---

Root Cause

Mutable shared state in a singleton in a concurrent environment.

More precisely:

• ConnectionOptions held request-specific data
• It was registered as a singleton
• It was mutated per request

This violates a core rule:

Singletons must be stateless or immutable.

---

Why This Is Hard to Catch

This issue didn’t show up in:

• Local development (low concurrency)
• Lambda (isolated execution)
• Unit tests (typically single-threaded)

It only surfaced under:

• Concurrent load
• Multi-threaded execution (ECS)

---

Fixes

Option 1: Use Scoped Lifetime

services.AddScoped<ConnectionOptions>();

Each request gets its own instance.

---

Option 2: Make It Immutable

public class ConnectionOptions
{
    public IReadOnlyDictionary<string, string> Headers { get; }

    public ConnectionOptions(Dictionary<string, string> headers)
    {
        Headers = headers;
    }
}

Create a new instance per request instead of mutating.

---

Option 3: Avoid Shared State (Preferred)

Pass headers explicitly:

await client.CallAsync(headers);

Or:

var request = new HttpRequestMessage();
request.Headers.Add("Authorization", token);

This removes shared mutable state entirely.

---

Key Takeaways

1. Execution Model Matters

Lambda and ECS behave differently:

• Lambda hides concurrency issues
• ECS exposes them

---

2. Singleton ≠ Safe

Singletons are:

• Shared globally
• Unsafe if mutable

Use only for:

• Stateless services
• Immutable configuration

---

3. Don’t Store Request Data in Singletons

If data changes per request:

• It should not live in a singleton

---

4. Concurrency Bugs Are Non-Deterministic

They:

• Are hard to reproduce
• Appear random
• Often show up only in production

---

Final Thought

This wasn’t a .NET 10 issue or an ECS issue.

It was a design flaw that only became visible when the runtime stopped masking it.

When migrating from Lambda to ECS (or any long-running service model), review:

• Singleton usage
• Shared mutable state
• Request-scoped data handling

That’s where subtle, high-impact bugs tend to hide.