The profiler output didn't lie: our Rust service was spending 78% of its CPU time in atomic operations. Not business logic. Not I/O. Atomic operations. Our "zero-cost abstractions" language was burning cycles on reference counting.
The culprit was innocent-looking code that every Rust tutorial teaches: Arc<Mutex<T>> for sharing state between threads. We'd followed all the best practices, avoided data races, made the borrow checker happy. But we'd also accidentally created a reference counting nightmare that was strangling our performance.
Sometimes the safest code is the slowest code.
The Pattern Everyone Uses
Here's the code that looked perfectly reasonable in code review:
use std::sync::{Arc, Mutex};
use std::thread;
#[derive(Clone)]
struct MetricsCollector {
counters: Arc<Mutex<HashMap<String, u64>>>,
}
impl MetricsCollector {
fn increment(&self, key: &str) {
let counters_clone = Arc::clone(&self.counters);
thread::spawn(move || {
let mut counters = counters_clone.lock().unwrap();
*counters.entry(key.to_string()).or_insert(0) += 1;
});
}
}Clean, thread-safe, follows Rust idioms. The borrow checker loved it. The CPU didn't.
Every call to increment() was cloning an Arc, spawning a thread, and fighting for the same mutex. At scale, this created three performance disasters:
- Refcount churn: Constant atomic increments/decrements on the Arc
- Lock contention: Hundreds of threads fighting for the same mutex
- Cache line bouncing: The atomic reference count ping-ponging between CPU cores
The Hidden Cost of Arc::clone()
Here's what most developers don't realize: Arc::clone() isn't free. Each clone increments an atomic reference counter, and each drop decrements it. Each call to Arc::clone() increments the reference count atomically, and when multiple threads frequently update the same atomic variable (the reference count), it can cause cache lines to bounce between CPU cores, leading to increased latency and reduced throughput.
Under heavy load, we were creating thousands of Arc clones per second. Each clone meant:
- An atomic increment (expensive)
- Cache invalidation across CPU cores
- Memory barriers for consistency
- Eventual atomic decrement when the clone dropped
The "cheap" clone was costing us more than the actual work.
The Benchmarks That Opened Our Eyes
We wrote a simple benchmark to measure the real cost:
// Version 1: Heavy Arc cloning
fn heavy_arc_usage() {
let data = Arc::new(Mutex::new(0u64));
let handles: Vec<_> = (0..1000).map(|_| {
let data_clone = Arc::clone(&data);
thread::spawn(move || {
let mut guard = data_clone.lock().unwrap();
*guard += 1;
})
}).collect();
for handle in handles {
handle.join().unwrap();
}
}
// Version 2: Single Arc, passed by reference
fn light_arc_usage() {
let data = Arc::new(Mutex::new(0u64));
let handles: Vec<_> = (0..1000).map(|_| {
let data_ref = &data;
thread::spawn(move || {
let mut guard = data_ref.lock().unwrap();
*guard += 1;
})
}).collect();
for handle in handles {
handle.join().unwrap();
}
}The results were shocking:
- Heavy Arc version: 847ms, 83% time in atomic operations
- Light Arc version: 156ms, 12% time in atomic operations
The excessive Arc cloning was making our code 5x slower.
The Mutex Contention Multiplier
But Arc cloning wasn't our only problem. The shared Mutex<HashMap> was creating a bottleneck where hundreds of threads waited for the same lock. Using Mutex can cause threads to block frequently, especially in high-contention scenarios. If threads spend too much time waiting for the lock, overall performance suffers.
Mutex operations take longer because each thread must acquire and release the lock, introducing contention and locking overhead. Our profiler showed threads spending 60% of their time just waiting for lock acquisition.
The Solutions That Actually Work
1. Reduce Arc Clones
Instead of cloning Arcs everywhere, we restructured to minimize clones:
// Before: Clone Arc in every function call
fn bad_pattern(shared: Arc<Mutex<Data>>) {
let shared_clone = Arc::clone(&shared);
thread::spawn(move || {
// work with shared_clone
});
}
// After: Pass Arc by reference where possible
fn good_pattern(shared: &Arc<Mutex<Data>>) {
thread::scope(|s| {
s.spawn(|| {
// work with shared directly
});
});
}2. Replace Mutex with Better Primitives
For simple counters, we ditched Arc<Mutex<u64>> for Arc<AtomicU64>:
// Before: Heavy mutex for simple counter
let counter = Arc::new(Mutex::new(0u64));
// After: Atomic operations
let counter = Arc::new(AtomicU64::new(0));
counter.fetch_add(1, Ordering::Relaxed);Atomic operations avoid locking overhead, leading to much faster increments, especially with many threads.
3. Shard the Data
For the HashMap, we replaced one heavily-contended mutex with multiple lightly-contended ones:
struct ShardedCounters {
shards: Vec<Mutex<HashMap<String, u64>>>,
}
impl ShardedCounters {
fn increment(&self, key: &str) {
let shard_idx = hash(key) % self.shards.len();
let mut shard = self.shards[shard_idx].lock().unwrap();
*shard.entry(key.to_string()).or_insert(0) += 1;
}
}This reduced contention by spreading load across multiple mutexes.
The Performance Results
After our optimizations:
- Latency: 95th percentile dropped from 2.3s to 180ms
- Throughput: Requests/second increased 6x
- CPU usage: Dropped from 95% to 34% under load
- Arc operations: From 78% to 8% of CPU time
The same hardware was suddenly handling 6x more traffic.
The Broader Lessons
1. Arc::clone() Isn't Free: Every clone means atomic operations and potential cache invalidation.
2. Shared Mutexes Don't Scale: One heavily-contended mutex is worse than multiple lightly-contended ones.
3. Atomic Types Beat Mutex for Simple Data: AtomicU64 is much faster than Mutex<u64> for counters.
4. Profile Before Optimizing: Our intuition about bottlenecks was completely wrong.
When Arc<Mutex> Is Actually Good
Don't avoid Arc<Mutex> entirely, it's great when:
- You need complex shared state (not just counters)
- Access patterns are read-heavy with infrequent writes
- You're not cloning Arcs excessively
- Lock contention is low
The Real Lesson
Rust's safety guarantees don't come with performance guarantees. The borrow checker ensures your code won't crash, but it can't prevent you from writing slow code.
Arc<Mutex> patterns can lead to performance bottlenecks, especially under high lock contention. The "safe" patterns that Rust tutorials teach work great for correctness but might not work great for performance.
Our production service taught us that sometimes the most dangerous performance bugs are hiding in the safest-looking code. The patterns that make Rust's borrow checker happy don't always make the CPU happy.
When everyone tells you that Rust gives you "zero-cost abstractions," remember: Arc isn't one of them. Every clone has a cost, and at scale, that cost adds up fast.